2024-03-29T11:22:55.391 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T11:22:55.392 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622722 --owner scheduled_amathuri@teuthology --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622722/orig.config.yaml 2024-03-29T11:22:55.426 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:22:55.504 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622722 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622722' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi072.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBnkRsNyU0uNAD+2aatvNpmr+GleVEkLFEE6m7MMj06nj5J8vftU4nbNo8dSce5aVYqvPCwgkVTxgpesC/Q1+bI= smithi089.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPxSfkA8KNtwlNRpzeShqCxAH0NIYQ6PieBELRqnTCPw6tYsU9YdIh3hFkW56jYhLxj9fCJLD06xQQkl7q+9VVY= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T11:22:55.504 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T11:22:55.504 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T11:22:55.505 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T11:22:55.506 INFO:teuthology.task.internal:Checking packages... 2024-03-29T11:22:55.530 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T11:22:55.530 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T11:22:55.530 INFO:teuthology.packaging:ref: None 2024-03-29T11:22:55.530 INFO:teuthology.packaging:tag: None 2024-03-29T11:22:55.530 INFO:teuthology.packaging:branch: main 2024-03-29T11:22:55.530 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:22:55.530 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T11:22:56.036 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T11:22:56.038 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T11:22:56.046 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T11:22:56.046 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T11:22:56.090 INFO:teuthology.task.internal:Saving configuration 2024-03-29T11:22:56.103 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T11:22:56.115 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:22:56.140 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi072.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622722', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:16:41.526377', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBnkRsNyU0uNAD+2aatvNpmr+GleVEkLFEE6m7MMj06nj5J8vftU4nbNo8dSce5aVYqvPCwgkVTxgpesC/Q1+bI='} 2024-03-29T11:22:56.161 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi089.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622722', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:16:41.527681', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPxSfkA8KNtwlNRpzeShqCxAH0NIYQ6PieBELRqnTCPw6tYsU9YdIh3hFkW56jYhLxj9fCJLD06xQQkl7q+9VVY='} 2024-03-29T11:22:56.161 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T11:22:56.167 INFO:teuthology.task.internal:roles: ubuntu@smithi072.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T11:22:56.167 INFO:teuthology.task.internal:roles: ubuntu@smithi089.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T11:22:56.167 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T11:22:56.252 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6da951e0d0>, signals=[15]) 2024-03-29T11:22:56.252 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T11:22:56.258 INFO:teuthology.task.internal:Opening connections... 2024-03-29T11:22:56.258 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi072.front.sepia.ceph.com 2024-03-29T11:22:56.260 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:22:56.353 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi089.front.sepia.ceph.com 2024-03-29T11:22:56.354 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:22:56.441 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T11:22:56.448 DEBUG:teuthology.orchestra.run.smithi072:> uname -m 2024-03-29T11:22:56.465 INFO:teuthology.orchestra.run.smithi072.stdout:x86_64 2024-03-29T11:22:56.466 DEBUG:teuthology.orchestra.run.smithi072:> cat /etc/os-release 2024-03-29T11:22:56.522 INFO:teuthology.orchestra.run.smithi072.stdout:NAME="CentOS Stream" 2024-03-29T11:22:56.522 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION="9" 2024-03-29T11:22:56.523 INFO:teuthology.orchestra.run.smithi072.stdout:ID="centos" 2024-03-29T11:22:56.523 INFO:teuthology.orchestra.run.smithi072.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:22:56.523 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION_ID="9" 2024-03-29T11:22:56.523 INFO:teuthology.orchestra.run.smithi072.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:22:56.523 INFO:teuthology.orchestra.run.smithi072.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:22:56.523 INFO:teuthology.orchestra.run.smithi072.stdout:ANSI_COLOR="0;31" 2024-03-29T11:22:56.523 INFO:teuthology.orchestra.run.smithi072.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:22:56.523 INFO:teuthology.orchestra.run.smithi072.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:22:56.523 INFO:teuthology.orchestra.run.smithi072.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:22:56.523 INFO:teuthology.orchestra.run.smithi072.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:22:56.523 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:22:56.523 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:22:56.524 INFO:teuthology.lock.ops:Updating smithi072.front.sepia.ceph.com on lock server 2024-03-29T11:22:56.539 DEBUG:teuthology.orchestra.run.smithi089:> uname -m 2024-03-29T11:22:56.556 INFO:teuthology.orchestra.run.smithi089.stdout:x86_64 2024-03-29T11:22:56.557 DEBUG:teuthology.orchestra.run.smithi089:> cat /etc/os-release 2024-03-29T11:22:56.613 INFO:teuthology.orchestra.run.smithi089.stdout:NAME="CentOS Stream" 2024-03-29T11:22:56.613 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION="9" 2024-03-29T11:22:56.613 INFO:teuthology.orchestra.run.smithi089.stdout:ID="centos" 2024-03-29T11:22:56.613 INFO:teuthology.orchestra.run.smithi089.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:22:56.613 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION_ID="9" 2024-03-29T11:22:56.613 INFO:teuthology.orchestra.run.smithi089.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:22:56.613 INFO:teuthology.orchestra.run.smithi089.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:22:56.613 INFO:teuthology.orchestra.run.smithi089.stdout:ANSI_COLOR="0;31" 2024-03-29T11:22:56.613 INFO:teuthology.orchestra.run.smithi089.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:22:56.613 INFO:teuthology.orchestra.run.smithi089.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:22:56.613 INFO:teuthology.orchestra.run.smithi089.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:22:56.613 INFO:teuthology.orchestra.run.smithi089.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:22:56.613 INFO:teuthology.orchestra.run.smithi089.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:22:56.613 INFO:teuthology.orchestra.run.smithi089.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:22:56.614 INFO:teuthology.lock.ops:Updating smithi089.front.sepia.ceph.com on lock server 2024-03-29T11:22:56.647 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T11:22:56.656 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T11:22:56.673 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T11:22:56.673 DEBUG:teuthology.orchestra.run.smithi072:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:22:56.676 DEBUG:teuthology.orchestra.run.smithi089:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:22:56.691 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T11:22:56.697 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T11:22:56.697 DEBUG:teuthology.orchestra.run.smithi072:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:22:56.733 DEBUG:teuthology.orchestra.run.smithi089:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:22:56.890 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T11:22:56.945 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T11:22:56.960 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T11:22:56.960 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T11:22:56.960 DEBUG:teuthology.orchestra.run.smithi072:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:22:56.961 DEBUG:teuthology.orchestra.run.smithi089:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:22:56.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:22:56.977 DEBUG:teuthology.orchestra.run.smithi089:> uname -r 2024-03-29T11:22:56.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:22:56.978 DEBUG:teuthology.orchestra.run.smithi072:> uname -r 2024-03-29T11:22:57.033 INFO:teuthology.orchestra.run.smithi089.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:22:57.034 INFO:teuthology.task.kernel:Running kernel on smithi089: 5.14.0-361.el9.x86_64 2024-03-29T11:22:57.034 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum install -y kernel 2024-03-29T11:22:57.035 INFO:teuthology.orchestra.run.smithi072.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:22:57.035 INFO:teuthology.task.kernel:Running kernel on smithi072: 5.14.0-361.el9.x86_64 2024-03-29T11:22:57.035 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum install -y kernel 2024-03-29T11:22:59.185 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-29T11:23:00.104 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-29T11:23:00.784 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS-9 - BaseOS 4.2 MB/s | 2.0 MB 00:00 2024-03-29T11:23:01.480 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS-9 - BaseOS 7.3 MB/s | 2.0 MB 00:00 2024-03-29T11:23:02.307 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - BaseOS 6.7 MB/s | 8.0 MB 00:01 2024-03-29T11:23:04.192 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-03-29T11:23:04.423 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - BaseOS 3.1 MB/s | 8.0 MB 00:02 2024-03-29T11:23:06.692 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-03-29T11:23:08.209 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-29T11:23:09.677 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - Extras packages 48 kB/s | 16 kB 00:00 2024-03-29T11:23:11.189 INFO:teuthology.orchestra.run.smithi089.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-29T11:23:16.145 INFO:teuthology.orchestra.run.smithi089.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-29T11:23:18.194 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - CRB 786 kB/s | 6.1 MB 00:07 2024-03-29T11:23:18.385 INFO:teuthology.orchestra.run.smithi089.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:23:18.385 INFO:teuthology.orchestra.run.smithi089.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:23:18.508 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-29T11:23:18.514 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-29T11:23:18.514 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-03-29T11:23:18.514 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-29T11:23:18.514 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-03-29T11:23:18.514 INFO:teuthology.orchestra.run.smithi089.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:23:18.514 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-03-29T11:23:18.514 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:23:18.514 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:23:18.514 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:23:18.514 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:23:18.514 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-03-29T11:23:18.515 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-29T11:23:18.515 INFO:teuthology.orchestra.run.smithi089.stdout:Install 4 Packages 2024-03-29T11:23:18.515 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:23:18.515 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 76 M 2024-03-29T11:23:18.515 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 123 M 2024-03-29T11:23:18.515 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-03-29T11:23:18.747 INFO:teuthology.orchestra.run.smithi089.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 829 kB/s | 192 kB 00:00 2024-03-29T11:23:19.597 INFO:teuthology.orchestra.run.smithi089.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 14 MB/s | 15 MB 00:01 2024-03-29T11:23:19.673 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - Extras packages 45 kB/s | 16 kB 00:00 2024-03-29T11:23:20.130 INFO:teuthology.orchestra.run.smithi089.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 21 MB/s | 34 MB 00:01 2024-03-29T11:23:20.681 INFO:teuthology.orchestra.run.smithi089.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 14 MB/s | 28 MB 00:01 2024-03-29T11:23:20.684 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:23:20.684 INFO:teuthology.orchestra.run.smithi089.stdout:Total 35 MB/s | 76 MB 00:02 2024-03-29T11:23:20.685 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-03-29T11:23:20.734 INFO:teuthology.orchestra.run.smithi072.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-03-29T11:23:21.060 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-03-29T11:23:21.060 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-03-29T11:23:21.887 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-03-29T11:23:21.888 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-03-29T11:23:24.175 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-03-29T11:23:24.773 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:23:24.865 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:23:25.693 INFO:teuthology.orchestra.run.smithi072.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-29T11:23:27.623 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:23:27.760 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:23:27.951 INFO:teuthology.orchestra.run.smithi072.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:23:27.952 INFO:teuthology.orchestra.run.smithi072.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:23:28.056 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-29T11:23:28.072 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-29T11:23:28.072 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2024-03-29T11:23:28.072 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-29T11:23:28.072 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-03-29T11:23:28.072 INFO:teuthology.orchestra.run.smithi072.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:23:28.072 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2024-03-29T11:23:28.072 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:23:28.072 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:23:28.072 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:23:28.072 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:23:28.072 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-03-29T11:23:28.072 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-29T11:23:28.072 INFO:teuthology.orchestra.run.smithi072.stdout:Install 4 Packages 2024-03-29T11:23:28.073 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:23:28.073 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 76 M 2024-03-29T11:23:28.073 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 123 M 2024-03-29T11:23:28.073 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-03-29T11:23:28.238 INFO:teuthology.orchestra.run.smithi072.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-03-29T11:23:28.822 INFO:teuthology.orchestra.run.smithi072.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 20 MB/s | 15 MB 00:00 2024-03-29T11:23:29.490 INFO:teuthology.orchestra.run.smithi072.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 24 MB/s | 34 MB 00:01 2024-03-29T11:23:30.182 INFO:teuthology.orchestra.run.smithi072.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 14 MB/s | 28 MB 00:01 2024-03-29T11:23:30.186 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:23:30.186 INFO:teuthology.orchestra.run.smithi072.stdout:Total 36 MB/s | 76 MB 00:02 2024-03-29T11:23:30.186 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-03-29T11:23:30.586 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-03-29T11:23:30.586 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-03-29T11:23:31.412 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-03-29T11:23:31.412 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-03-29T11:23:32.482 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:23:32.518 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:23:33.704 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-03-29T11:23:34.351 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:23:34.455 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:23:37.208 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:23:37.266 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:23:37.399 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:23:42.104 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:23:42.144 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:23:46.851 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:24:06.994 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:24:06.994 INFO:teuthology.orchestra.run.smithi089.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:24:06.994 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:24:07.038 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:24:07.374 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:24:07.374 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:24:07.375 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:24:07.375 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:24:07.814 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:24:07.814 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:24:07.814 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-03-29T11:24:07.814 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:24:07.814 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:24:07.814 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:24:07.814 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:24:07.814 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:24:07.814 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-29T11:24:08.147 DEBUG:teuthology.orchestra.run.smithi089:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:24:08.250 INFO:teuthology.orchestra.run.smithi089.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:24:08.251 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:24:08.251 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:24:08.251 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:24:08.251 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:24:08.251 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T11:24:08.251 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi089.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:24:08.251 DEBUG:teuthology.orchestra.run.smithi089:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:24:08.331 INFO:teuthology.orchestra.run.smithi089.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:24:08.332 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:24:08.332 DEBUG:teuthology.orchestra.run.smithi089:> sudo rpm -qi grub2-tools 2024-03-29T11:24:08.371 INFO:teuthology.orchestra.run.smithi089.stdout:Name : grub2-tools 2024-03-29T11:24:08.371 INFO:teuthology.orchestra.run.smithi089.stdout:Epoch : 1 2024-03-29T11:24:08.371 INFO:teuthology.orchestra.run.smithi089.stdout:Version : 2.06 2024-03-29T11:24:08.371 INFO:teuthology.orchestra.run.smithi089.stdout:Release : 68.el9 2024-03-29T11:24:08.372 INFO:teuthology.orchestra.run.smithi089.stdout:Architecture: x86_64 2024-03-29T11:24:08.372 INFO:teuthology.orchestra.run.smithi089.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:24:08.372 INFO:teuthology.orchestra.run.smithi089.stdout:Group : Unspecified 2024-03-29T11:24:08.372 INFO:teuthology.orchestra.run.smithi089.stdout:Size : 8264254 2024-03-29T11:24:08.372 INFO:teuthology.orchestra.run.smithi089.stdout:License : GPLv3+ 2024-03-29T11:24:08.372 INFO:teuthology.orchestra.run.smithi089.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:24:08.372 INFO:teuthology.orchestra.run.smithi089.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:24:08.372 INFO:teuthology.orchestra.run.smithi089.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:24:08.372 INFO:teuthology.orchestra.run.smithi089.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:24:08.372 INFO:teuthology.orchestra.run.smithi089.stdout:Packager : builder@centos.org 2024-03-29T11:24:08.372 INFO:teuthology.orchestra.run.smithi089.stdout:Vendor : CentOS 2024-03-29T11:24:08.372 INFO:teuthology.orchestra.run.smithi089.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:24:08.372 INFO:teuthology.orchestra.run.smithi089.stdout:Summary : Support tools for GRUB. 2024-03-29T11:24:08.372 INFO:teuthology.orchestra.run.smithi089.stdout:Description : 2024-03-29T11:24:08.372 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:24:08.373 INFO:teuthology.orchestra.run.smithi089.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:24:08.373 INFO:teuthology.orchestra.run.smithi089.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:24:08.373 INFO:teuthology.orchestra.run.smithi089.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:24:08.373 INFO:teuthology.orchestra.run.smithi089.stdout:hardware devices. 2024-03-29T11:24:08.373 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:24:08.373 INFO:teuthology.orchestra.run.smithi089.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:24:08.374 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:24:08.374 INFO:teuthology.task.kernel:Updating grub on smithi089 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:24:08.374 DEBUG:teuthology.orchestra.run.smithi089:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:24:09.004 INFO:teuthology.orchestra.run.smithi089.stderr:Generating grub configuration file ... 2024-03-29T11:24:10.309 INFO:teuthology.orchestra.run.smithi089.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:24:10.358 INFO:teuthology.orchestra.run.smithi089.stderr:done 2024-03-29T11:24:10.361 DEBUG:teuthology.orchestra.run.smithi089:> mktemp 2024-03-29T11:24:10.376 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/tmp.2Ej0zgrIXc 2024-03-29T11:24:10.376 DEBUG:teuthology.orchestra.run.smithi089:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.2Ej0zgrIXc 2024-03-29T11:24:10.445 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 0666 /tmp/tmp.2Ej0zgrIXc 2024-03-29T11:24:10.585 DEBUG:teuthology.orchestra.remote:smithi089:/tmp/tmp.2Ej0zgrIXc is 6KB 2024-03-29T11:24:10.633 DEBUG:teuthology.orchestra.run.smithi089:> rm -fr /tmp/tmp.2Ej0zgrIXc 2024-03-29T11:24:10.647 DEBUG:teuthology.orchestra.run.smithi089:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:24:10.712 INFO:teuthology.orchestra.run.smithi089.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:24:10.712 INFO:teuthology.orchestra.run.smithi089.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:24:10.712 INFO:teuthology.orchestra.run.smithi089.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:24:10.712 INFO:teuthology.orchestra.run.smithi089.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:24:10.713 DEBUG:teuthology.orchestra.run.smithi089:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:24:10.869 DEBUG:teuthology.orchestra.run.smithi089:> sudo shutdown -r now 2024-03-29T11:24:18.280 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:24:18.280 INFO:teuthology.orchestra.run.smithi072.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:24:18.280 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:24:18.322 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:24:18.681 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:24:18.681 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:24:18.681 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:24:18.681 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:24:19.092 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:24:19.092 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:24:19.092 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-03-29T11:24:19.092 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:24:19.092 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:24:19.092 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:24:19.092 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:24:19.092 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:24:19.092 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-29T11:24:19.533 DEBUG:teuthology.orchestra.run.smithi072:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:24:19.661 INFO:teuthology.orchestra.run.smithi072.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:24:19.661 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:24:19.661 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:24:19.661 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:24:19.661 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:24:19.661 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T11:24:19.661 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi072.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:24:19.662 DEBUG:teuthology.orchestra.run.smithi072:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:24:19.742 INFO:teuthology.orchestra.run.smithi072.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:24:19.742 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:24:19.742 DEBUG:teuthology.orchestra.run.smithi072:> sudo rpm -qi grub2-tools 2024-03-29T11:24:19.778 INFO:teuthology.orchestra.run.smithi072.stdout:Name : grub2-tools 2024-03-29T11:24:19.779 INFO:teuthology.orchestra.run.smithi072.stdout:Epoch : 1 2024-03-29T11:24:19.779 INFO:teuthology.orchestra.run.smithi072.stdout:Version : 2.06 2024-03-29T11:24:19.779 INFO:teuthology.orchestra.run.smithi072.stdout:Release : 68.el9 2024-03-29T11:24:19.779 INFO:teuthology.orchestra.run.smithi072.stdout:Architecture: x86_64 2024-03-29T11:24:19.779 INFO:teuthology.orchestra.run.smithi072.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:24:19.779 INFO:teuthology.orchestra.run.smithi072.stdout:Group : Unspecified 2024-03-29T11:24:19.779 INFO:teuthology.orchestra.run.smithi072.stdout:Size : 8264254 2024-03-29T11:24:19.779 INFO:teuthology.orchestra.run.smithi072.stdout:License : GPLv3+ 2024-03-29T11:24:19.779 INFO:teuthology.orchestra.run.smithi072.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:24:19.779 INFO:teuthology.orchestra.run.smithi072.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:24:19.779 INFO:teuthology.orchestra.run.smithi072.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:24:19.779 INFO:teuthology.orchestra.run.smithi072.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:24:19.779 INFO:teuthology.orchestra.run.smithi072.stdout:Packager : builder@centos.org 2024-03-29T11:24:19.779 INFO:teuthology.orchestra.run.smithi072.stdout:Vendor : CentOS 2024-03-29T11:24:19.780 INFO:teuthology.orchestra.run.smithi072.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:24:19.780 INFO:teuthology.orchestra.run.smithi072.stdout:Summary : Support tools for GRUB. 2024-03-29T11:24:19.780 INFO:teuthology.orchestra.run.smithi072.stdout:Description : 2024-03-29T11:24:19.780 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:24:19.780 INFO:teuthology.orchestra.run.smithi072.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:24:19.780 INFO:teuthology.orchestra.run.smithi072.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:24:19.780 INFO:teuthology.orchestra.run.smithi072.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:24:19.782 INFO:teuthology.orchestra.run.smithi072.stdout:hardware devices. 2024-03-29T11:24:19.782 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:24:19.782 INFO:teuthology.orchestra.run.smithi072.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:24:19.783 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:24:19.783 INFO:teuthology.task.kernel:Updating grub on smithi072 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:24:19.783 DEBUG:teuthology.orchestra.run.smithi072:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:24:20.448 INFO:teuthology.orchestra.run.smithi072.stderr:Generating grub configuration file ... 2024-03-29T11:24:21.959 INFO:teuthology.orchestra.run.smithi072.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:24:21.976 INFO:teuthology.orchestra.run.smithi072.stderr:done 2024-03-29T11:24:21.978 DEBUG:teuthology.orchestra.run.smithi072:> mktemp 2024-03-29T11:24:22.013 INFO:teuthology.orchestra.run.smithi072.stdout:/tmp/tmp.LDCQm7Rx7W 2024-03-29T11:24:22.014 DEBUG:teuthology.orchestra.run.smithi072:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.LDCQm7Rx7W 2024-03-29T11:24:22.082 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 0666 /tmp/tmp.LDCQm7Rx7W 2024-03-29T11:24:22.223 DEBUG:teuthology.orchestra.remote:smithi072:/tmp/tmp.LDCQm7Rx7W is 6KB 2024-03-29T11:24:22.271 DEBUG:teuthology.orchestra.run.smithi072:> rm -fr /tmp/tmp.LDCQm7Rx7W 2024-03-29T11:24:22.285 DEBUG:teuthology.orchestra.run.smithi072:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:24:22.350 INFO:teuthology.orchestra.run.smithi072.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:24:22.350 INFO:teuthology.orchestra.run.smithi072.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:24:22.350 INFO:teuthology.orchestra.run.smithi072.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:24:22.350 INFO:teuthology.orchestra.run.smithi072.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:24:22.351 DEBUG:teuthology.orchestra.run.smithi072:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:24:22.505 DEBUG:teuthology.orchestra.run.smithi072:> sudo shutdown -r now 2024-03-29T11:24:40.886 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T11:24:40.887 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-03-29T11:24:40.887 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:24:52.518 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T11:24:52.519 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-03-29T11:24:52.519 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:25:40.937 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T11:25:49.946 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-03-29T11:25:49.947 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:25:51.506 DEBUG:teuthology.orchestra.run.smithi089:> true 2024-03-29T11:25:52.063 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-03-29T11:25:52.063 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:25:52.063 DEBUG:teuthology.orchestra.run.smithi089:> uname -r 2024-03-29T11:25:52.077 INFO:teuthology.orchestra.run.smithi089.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:25:52.077 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:25:52.077 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:25:52.077 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:25:52.521 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T11:25:53.079 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T11:25:53.079 DEBUG:teuthology.orchestra.run.smithi089:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:25:53.146 INFO:teuthology.orchestra.run.smithi089.stdout:ttyS1 2024-03-29T11:25:53.187 DEBUG:teuthology.parallel:result is None 2024-03-29T11:26:01.522 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-03-29T11:26:01.523 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:26:09.355 DEBUG:teuthology.orchestra.run.smithi072:> true 2024-03-29T11:26:09.825 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-03-29T11:26:09.825 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:26:09.825 DEBUG:teuthology.orchestra.run.smithi072:> uname -r 2024-03-29T11:26:09.839 INFO:teuthology.orchestra.run.smithi072.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:26:09.839 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:26:09.839 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:26:09.840 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:26:10.841 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T11:26:10.841 DEBUG:teuthology.orchestra.run.smithi072:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:26:10.975 INFO:teuthology.orchestra.run.smithi072.stdout:ttyS1 2024-03-29T11:26:11.031 DEBUG:teuthology.parallel:result is None 2024-03-29T11:26:11.031 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T11:26:11.039 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T11:26:11.039 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:26:11.041 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:26:11.060 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T11:26:11.084 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T11:26:11.117 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T11:26:11.118 DEBUG:teuthology.orchestra.run.smithi072:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:26:11.120 DEBUG:teuthology.orchestra.run.smithi089:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:26:11.274 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T11:26:11.281 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T11:26:11.282 DEBUG:teuthology.orchestra.run.smithi072:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:26:11.284 DEBUG:teuthology.orchestra.run.smithi089:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:26:11.318 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:26:11.322 INFO:teuthology.orchestra.run.smithi072.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:26:11.331 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:26:11.335 INFO:teuthology.orchestra.run.smithi072.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:26:11.336 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T11:26:11.343 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T11:26:11.343 DEBUG:teuthology.orchestra.run.smithi072:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:26:11.376 DEBUG:teuthology.orchestra.run.smithi089:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:26:11.405 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T11:26:11.415 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T11:26:11.415 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:26:11.439 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:26:11.461 DEBUG:teuthology.orchestra.run.smithi072:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:26:11.532 DEBUG:teuthology.orchestra.run.smithi072:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:26:11.583 DEBUG:teuthology.orchestra.run.smithi072:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:26:11.661 DEBUG:teuthology.orchestra.run.smithi072:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:26:11.707 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-29T11:26:11.707 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:26:11.774 DEBUG:teuthology.orchestra.run.smithi089:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:26:11.815 DEBUG:teuthology.orchestra.run.smithi089:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:26:11.873 DEBUG:teuthology.orchestra.run.smithi089:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:26:11.954 DEBUG:teuthology.orchestra.run.smithi089:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:26:12.000 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-29T11:26:12.000 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:26:12.068 DEBUG:teuthology.orchestra.run.smithi072:> sudo service rsyslog restart 2024-03-29T11:26:12.071 DEBUG:teuthology.orchestra.run.smithi089:> sudo service rsyslog restart 2024-03-29T11:26:12.127 INFO:teuthology.orchestra.run.smithi072.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:26:12.157 INFO:teuthology.orchestra.run.smithi089.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:26:12.669 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T11:26:12.676 INFO:teuthology.task.internal:Starting timer... 2024-03-29T11:26:12.676 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T11:26:12.721 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T11:26:12.753 DEBUG:teuthology.orchestra.run.smithi072:> sudo service auditd rotate 2024-03-29T11:26:12.854 INFO:teuthology.orchestra.run.smithi072.stdout:Rotating logs: 2024-03-29T11:26:12.856 DEBUG:teuthology.orchestra.run.smithi089:> sudo service auditd rotate 2024-03-29T11:26:12.915 INFO:teuthology.orchestra.run.smithi089.stdout:Rotating logs: 2024-03-29T11:26:12.916 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T11:26:12.916 DEBUG:teuthology.orchestra.run.smithi072:> /usr/sbin/getenforce 2024-03-29T11:26:12.940 INFO:teuthology.orchestra.run.smithi072.stdout:Permissive 2024-03-29T11:26:12.940 DEBUG:teuthology.orchestra.run.smithi089:> /usr/sbin/getenforce 2024-03-29T11:26:12.961 INFO:teuthology.orchestra.run.smithi089.stdout:Permissive 2024-03-29T11:26:12.962 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi072.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi089.front.sepia.ceph.com': 'permissive'} 2024-03-29T11:26:12.962 DEBUG:teuthology.orchestra.run.smithi072:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:26:13.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:26:13.007 DEBUG:teuthology.orchestra.run.smithi089:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:26:13.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:26:13.037 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T11:26:13.037 DEBUG:teuthology.orchestra.run.smithi072:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:26:13.084 DEBUG:teuthology.orchestra.run.smithi089:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:26:13.123 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T11:26:13.134 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T11:26:13.153 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T11:26:13.154 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi072.front.sepia.ceph.com,smithi089.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T11:30:40.437 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi072.front.sepia.ceph.com'), Remote(name='ubuntu@smithi089.front.sepia.ceph.com')] 2024-03-29T11:30:40.439 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-03-29T11:30:40.440 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:30:40.536 DEBUG:teuthology.orchestra.run.smithi072:> true 2024-03-29T11:30:40.618 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-03-29T11:30:40.618 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-03-29T11:30:40.619 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:30:40.707 DEBUG:teuthology.orchestra.run.smithi089:> true 2024-03-29T11:30:40.780 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-03-29T11:30:40.781 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T11:30:40.791 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T11:30:40.792 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:30:40.792 DEBUG:teuthology.orchestra.run.smithi072:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:30:40.794 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:30:40.795 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:30:40.823 INFO:teuthology.orchestra.run.smithi072.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:30:40.840 INFO:teuthology.orchestra.run.smithi072.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:30:40.851 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:30:40.867 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:30:40.868 INFO:teuthology.orchestra.run.smithi072.stderr:sudo: ntpd: command not found 2024-03-29T11:30:40.882 INFO:teuthology.orchestra.run.smithi072.stdout:506 Cannot talk to daemon 2024-03-29T11:30:40.896 INFO:teuthology.orchestra.run.smithi089.stderr:sudo: ntpd: command not found 2024-03-29T11:30:40.898 INFO:teuthology.orchestra.run.smithi072.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:30:40.910 INFO:teuthology.orchestra.run.smithi089.stdout:506 Cannot talk to daemon 2024-03-29T11:30:40.915 INFO:teuthology.orchestra.run.smithi072.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:30:40.926 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:30:40.942 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:30:40.969 INFO:teuthology.orchestra.run.smithi072.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:30:40.972 INFO:teuthology.orchestra.run.smithi072.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:30:40.972 INFO:teuthology.orchestra.run.smithi072.stdout:=============================================================================== 2024-03-29T11:30:40.972 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:30:40.973 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:30:40.973 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:30:40.973 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:30:40.993 INFO:teuthology.orchestra.run.smithi089.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:30:40.996 INFO:teuthology.orchestra.run.smithi089.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:30:40.996 INFO:teuthology.orchestra.run.smithi089.stdout:=============================================================================== 2024-03-29T11:30:40.996 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:30:40.996 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:30:40.996 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:30:40.996 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:30:40.997 INFO:teuthology.run_tasks:Running task install... 2024-03-29T11:30:41.007 DEBUG:teuthology.task.install:project ceph 2024-03-29T11:30:41.007 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T11:30:41.007 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T11:30:41.007 INFO:teuthology.task.install:Using flavor: default 2024-03-29T11:30:41.013 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T11:30:41.013 INFO:teuthology.task.install:extra packages: [] 2024-03-29T11:30:41.013 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:30:41.014 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:30:41.016 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:30:41.016 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:30:41.157 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:30:41.157 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:30:41.164 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:30:41.164 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:30:41.353 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:30:41.353 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-29T11:30:41.353 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:30:41.382 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:30:41.382 DEBUG:teuthology.orchestra.run.smithi072:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:30:41.449 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:30:41.449 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-29T11:30:41.449 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:30:41.451 DEBUG:teuthology.orchestra.run.smithi072:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:30:41.479 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:30:41.479 DEBUG:teuthology.orchestra.run.smithi089:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:30:41.531 DEBUG:teuthology.orchestra.run.smithi072:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:30:41.547 DEBUG:teuthology.orchestra.run.smithi089:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:30:41.598 INFO:teuthology.orchestra.run.smithi072.stdout:check_obsoletes = 1 2024-03-29T11:30:41.599 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum clean all 2024-03-29T11:30:41.627 DEBUG:teuthology.orchestra.run.smithi089:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:30:41.694 INFO:teuthology.orchestra.run.smithi089.stdout:check_obsoletes = 1 2024-03-29T11:30:41.695 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum clean all 2024-03-29T11:30:41.835 INFO:teuthology.orchestra.run.smithi072.stdout:74 files removed 2024-03-29T11:30:41.862 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-radosgw 2024-03-29T11:30:41.938 INFO:teuthology.orchestra.run.smithi089.stdout:74 files removed 2024-03-29T11:30:41.965 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-radosgw 2024-03-29T11:30:42.956 INFO:teuthology.orchestra.run.smithi072.stdout:ceph packages for x86_64 103 kB/s | 82 kB 00:00 2024-03-29T11:30:43.380 INFO:teuthology.orchestra.run.smithi089.stdout:ceph packages for x86_64 73 kB/s | 82 kB 00:01 2024-03-29T11:30:43.524 INFO:teuthology.orchestra.run.smithi072.stdout:ceph noarch packages 19 kB/s | 10 kB 00:00 2024-03-29T11:30:44.131 INFO:teuthology.orchestra.run.smithi089.stdout:ceph noarch packages 14 kB/s | 10 kB 00:00 2024-03-29T11:30:44.278 INFO:teuthology.orchestra.run.smithi072.stdout:ceph source packages 2.7 kB/s | 1.9 kB 00:00 2024-03-29T11:30:44.815 INFO:teuthology.orchestra.run.smithi089.stdout:ceph source packages 2.9 kB/s | 1.9 kB 00:00 2024-03-29T11:30:44.820 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-29T11:30:45.767 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS-9 - AppStream 7.3 MB/s | 6.7 MB 00:00 2024-03-29T11:30:46.421 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS-9 - BaseOS 4.2 MB/s | 2.0 MB 00:00 2024-03-29T11:30:47.267 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS-9 - BaseOS 5.1 MB/s | 2.0 MB 00:00 2024-03-29T11:30:47.789 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - BaseOS 7.7 MB/s | 8.0 MB 00:01 2024-03-29T11:30:56.683 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - BaseOS 905 kB/s | 8.0 MB 00:09 2024-03-29T11:30:58.650 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-03-29T11:31:04.108 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:15 2024-03-29T11:31:08.135 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-29T11:31:10.192 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - CRB 779 kB/s | 6.1 MB 00:08 2024-03-29T11:31:10.480 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - Extras packages 13 kB/s | 16 kB 00:01 2024-03-29T11:31:12.026 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 16 kB 00:00 2024-03-29T11:31:12.630 INFO:teuthology.orchestra.run.smithi072.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:02 2024-03-29T11:31:14.694 INFO:teuthology.orchestra.run.smithi089.stdout:Extra Packages for Enterprise Linux 8.2 MB/s | 21 MB 00:02 2024-03-29T11:31:17.518 INFO:teuthology.orchestra.run.smithi072.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-03-29T11:31:19.828 INFO:teuthology.orchestra.run.smithi089.stdout:lab-extras 23 kB/s | 1.7 kB 00:00 2024-03-29T11:31:20.193 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-29T11:31:20.212 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-29T11:31:20.213 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2024-03-29T11:31:20.213 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-29T11:31:20.213 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-03-29T11:31:20.213 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:31:20.213 INFO:teuthology.orchestra.run.smithi072.stdout:Upgrading: 2024-03-29T11:31:20.213 INFO:teuthology.orchestra.run.smithi072.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:31:20.213 INFO:teuthology.orchestra.run.smithi072.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:31:20.213 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2024-03-29T11:31:20.213 INFO:teuthology.orchestra.run.smithi072.stdout: boost-program-options 2024-03-29T11:31:20.213 INFO:teuthology.orchestra.run.smithi072.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:31:20.213 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:31:20.213 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:31:20.214 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:31:20.214 INFO:teuthology.orchestra.run.smithi072.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:31:20.214 INFO:teuthology.orchestra.run.smithi072.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:31:20.214 INFO:teuthology.orchestra.run.smithi072.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:31:20.214 INFO:teuthology.orchestra.run.smithi072.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:31:20.214 INFO:teuthology.orchestra.run.smithi072.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:31:20.214 INFO:teuthology.orchestra.run.smithi072.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:31:20.214 INFO:teuthology.orchestra.run.smithi072.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:31:20.214 INFO:teuthology.orchestra.run.smithi072.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:31:20.214 INFO:teuthology.orchestra.run.smithi072.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:31:20.214 INFO:teuthology.orchestra.run.smithi072.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:31:20.214 INFO:teuthology.orchestra.run.smithi072.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:31:20.214 INFO:teuthology.orchestra.run.smithi072.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:31:20.215 INFO:teuthology.orchestra.run.smithi072.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:31:20.215 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ceph-argparse 2024-03-29T11:31:20.215 INFO:teuthology.orchestra.run.smithi072.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:31:20.215 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:31:20.215 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:31:20.215 INFO:teuthology.orchestra.run.smithi072.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:31:20.215 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:31:20.215 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:31:20.215 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:31:20.215 INFO:teuthology.orchestra.run.smithi072.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:31:20.215 INFO:teuthology.orchestra.run.smithi072.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:31:20.215 INFO:teuthology.orchestra.run.smithi072.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:31:20.216 INFO:teuthology.orchestra.run.smithi072.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:31:20.216 INFO:teuthology.orchestra.run.smithi072.stdout:Installing weak dependencies: 2024-03-29T11:31:20.216 INFO:teuthology.orchestra.run.smithi072.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:31:20.216 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:31:20.216 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-03-29T11:31:20.216 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-29T11:31:20.216 INFO:teuthology.orchestra.run.smithi072.stdout:Install 30 Packages 2024-03-29T11:31:20.216 INFO:teuthology.orchestra.run.smithi072.stdout:Upgrade 2 Packages 2024-03-29T11:31:20.216 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:31:20.217 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 58 M 2024-03-29T11:31:20.217 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-03-29T11:31:22.459 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-29T11:31:22.472 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-29T11:31:22.472 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-03-29T11:31:22.472 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-29T11:31:22.472 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-03-29T11:31:22.472 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:31:22.472 INFO:teuthology.orchestra.run.smithi089.stdout:Upgrading: 2024-03-29T11:31:22.472 INFO:teuthology.orchestra.run.smithi089.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:31:22.473 INFO:teuthology.orchestra.run.smithi089.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:31:22.473 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-03-29T11:31:22.473 INFO:teuthology.orchestra.run.smithi089.stdout: boost-program-options 2024-03-29T11:31:22.473 INFO:teuthology.orchestra.run.smithi089.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:31:22.473 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:31:22.473 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:31:22.473 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:31:22.473 INFO:teuthology.orchestra.run.smithi089.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:31:22.473 INFO:teuthology.orchestra.run.smithi089.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:31:22.473 INFO:teuthology.orchestra.run.smithi089.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:31:22.473 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:31:22.473 INFO:teuthology.orchestra.run.smithi089.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:31:22.473 INFO:teuthology.orchestra.run.smithi089.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:31:22.474 INFO:teuthology.orchestra.run.smithi089.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:31:22.474 INFO:teuthology.orchestra.run.smithi089.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:31:22.474 INFO:teuthology.orchestra.run.smithi089.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:31:22.474 INFO:teuthology.orchestra.run.smithi089.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:31:22.474 INFO:teuthology.orchestra.run.smithi089.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:31:22.474 INFO:teuthology.orchestra.run.smithi089.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:31:22.474 INFO:teuthology.orchestra.run.smithi089.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:31:22.474 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-argparse 2024-03-29T11:31:22.474 INFO:teuthology.orchestra.run.smithi089.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:31:22.474 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:31:22.474 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:31:22.475 INFO:teuthology.orchestra.run.smithi089.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:31:22.475 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:31:22.475 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:31:22.475 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:31:22.475 INFO:teuthology.orchestra.run.smithi089.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:31:22.475 INFO:teuthology.orchestra.run.smithi089.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:31:22.475 INFO:teuthology.orchestra.run.smithi089.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:31:22.475 INFO:teuthology.orchestra.run.smithi089.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:31:22.475 INFO:teuthology.orchestra.run.smithi089.stdout:Installing weak dependencies: 2024-03-29T11:31:22.475 INFO:teuthology.orchestra.run.smithi089.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:31:22.475 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:31:22.476 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-03-29T11:31:22.476 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-29T11:31:22.476 INFO:teuthology.orchestra.run.smithi089.stdout:Install 30 Packages 2024-03-29T11:31:22.476 INFO:teuthology.orchestra.run.smithi089.stdout:Upgrade 2 Packages 2024-03-29T11:31:22.476 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:31:22.476 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 58 M 2024-03-29T11:31:22.476 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-03-29T11:31:24.172 INFO:teuthology.orchestra.run.smithi072.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 1.5 MB/s | 5.3 MB 00:03 2024-03-29T11:31:24.272 INFO:teuthology.orchestra.run.smithi072.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 251 kB/s | 25 kB 00:00 2024-03-29T11:31:24.779 INFO:teuthology.orchestra.run.smithi089.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 2.6 MB/s | 5.3 MB 00:02 2024-03-29T11:31:24.881 INFO:teuthology.orchestra.run.smithi072.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 1.2 MB/s | 742 kB 00:00 2024-03-29T11:31:24.888 INFO:teuthology.orchestra.run.smithi089.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 233 kB/s | 25 kB 00:00 2024-03-29T11:31:25.280 INFO:teuthology.orchestra.run.smithi089.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 1.8 MB/s | 742 kB 00:00 2024-03-29T11:31:25.340 INFO:teuthology.orchestra.run.smithi072.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 1.1 MB/s | 498 kB 00:00 2024-03-29T11:31:25.472 INFO:teuthology.orchestra.run.smithi089.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 2.5 MB/s | 498 kB 00:00 2024-03-29T11:31:27.139 INFO:teuthology.orchestra.run.smithi089.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 3.1 MB/s | 5.1 MB 00:01 2024-03-29T11:31:27.255 INFO:teuthology.orchestra.run.smithi089.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 387 kB/s | 45 kB 00:00 2024-03-29T11:31:27.364 INFO:teuthology.orchestra.run.smithi089.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.2 MB/s | 131 kB 00:00 2024-03-29T11:31:27.481 INFO:teuthology.orchestra.run.smithi089.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 1.4 MB/s | 162 kB 00:00 2024-03-29T11:31:28.423 INFO:teuthology.orchestra.run.smithi089.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 340 kB/s | 320 kB 00:00 2024-03-29T11:31:29.190 INFO:teuthology.orchestra.run.smithi089.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 388 kB/s | 298 kB 00:00 2024-03-29T11:31:29.457 INFO:teuthology.orchestra.run.smithi089.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 373 kB/s | 99 kB 00:00 2024-03-29T11:31:29.565 INFO:teuthology.orchestra.run.smithi072.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 1.2 MB/s | 5.1 MB 00:04 2024-03-29T11:31:29.615 INFO:teuthology.orchestra.run.smithi089.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 670 kB/s | 106 kB 00:00 2024-03-29T11:31:29.658 INFO:teuthology.orchestra.run.smithi089.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-29T11:31:29.658 INFO:teuthology.orchestra.run.smithi072.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 493 kB/s | 45 kB 00:00 2024-03-29T11:31:29.699 INFO:teuthology.orchestra.run.smithi089.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-29T11:31:29.767 INFO:teuthology.orchestra.run.smithi089.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 9.6 MB/s | 661 kB 00:00 2024-03-29T11:31:29.808 INFO:teuthology.orchestra.run.smithi089.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.7 MB/s | 282 kB 00:00 2024-03-29T11:31:29.833 INFO:teuthology.orchestra.run.smithi072.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 749 kB/s | 131 kB 00:00 2024-03-29T11:31:29.883 INFO:teuthology.orchestra.run.smithi089.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 549 kB/s | 41 kB 00:00 2024-03-29T11:31:29.926 INFO:teuthology.orchestra.run.smithi089.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 5.1 MB/s | 220 kB 00:00 2024-03-29T11:31:29.968 INFO:teuthology.orchestra.run.smithi089.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 840 kB/s | 35 kB 00:00 2024-03-29T11:31:30.000 INFO:teuthology.orchestra.run.smithi072.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 970 kB/s | 162 kB 00:00 2024-03-29T11:31:30.017 INFO:teuthology.orchestra.run.smithi089.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 932 kB/s | 46 kB 00:00 2024-03-29T11:31:30.309 INFO:teuthology.orchestra.run.smithi072.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 1.0 MB/s | 320 kB 00:00 2024-03-29T11:31:30.585 INFO:teuthology.orchestra.run.smithi089.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 545 kB/s | 309 kB 00:00 2024-03-29T11:31:30.610 INFO:teuthology.orchestra.run.smithi072.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 992 kB/s | 298 kB 00:00 2024-03-29T11:31:30.860 INFO:teuthology.orchestra.run.smithi072.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 397 kB/s | 99 kB 00:00 2024-03-29T11:31:30.969 INFO:teuthology.orchestra.run.smithi072.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 975 kB/s | 106 kB 00:00 2024-03-29T11:31:31.018 INFO:teuthology.orchestra.run.smithi089.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-03-29T11:31:31.019 INFO:teuthology.orchestra.run.smithi072.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-29T11:31:31.070 INFO:teuthology.orchestra.run.smithi072.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 881 kB/s | 44 kB 00:00 2024-03-29T11:31:31.135 INFO:teuthology.orchestra.run.smithi089.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 223 kB/s | 26 kB 00:00 2024-03-29T11:31:31.170 INFO:teuthology.orchestra.run.smithi072.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 6.5 MB/s | 661 kB 00:00 2024-03-29T11:31:31.228 INFO:teuthology.orchestra.run.smithi089.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 530 kB/s | 49 kB 00:00 2024-03-29T11:31:31.231 INFO:teuthology.orchestra.run.smithi072.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.6 MB/s | 282 kB 00:00 2024-03-29T11:31:31.281 INFO:teuthology.orchestra.run.smithi072.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 819 kB/s | 41 kB 00:00 2024-03-29T11:31:31.344 INFO:teuthology.orchestra.run.smithi089.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 577 kB/s | 67 kB 00:00 2024-03-29T11:31:31.348 INFO:teuthology.orchestra.run.smithi072.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 3.2 MB/s | 220 kB 00:00 2024-03-29T11:31:31.408 INFO:teuthology.orchestra.run.smithi072.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 593 kB/s | 35 kB 00:00 2024-03-29T11:31:31.475 INFO:teuthology.orchestra.run.smithi072.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 693 kB/s | 46 kB 00:00 2024-03-29T11:31:31.494 INFO:teuthology.orchestra.run.smithi089.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.5 MB/s | 838 kB 00:00 2024-03-29T11:31:31.619 INFO:teuthology.orchestra.run.smithi089.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.5 MB/s | 192 kB 00:00 2024-03-29T11:31:31.795 INFO:teuthology.orchestra.run.smithi089.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 9.0 MB/s | 1.6 MB 00:00 2024-03-29T11:31:32.175 INFO:teuthology.orchestra.run.smithi072.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 441 kB/s | 309 kB 00:00 2024-03-29T11:31:32.726 INFO:teuthology.orchestra.run.smithi072.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 8.0 MB/s | 4.4 MB 00:00 2024-03-29T11:31:32.834 INFO:teuthology.orchestra.run.smithi072.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 240 kB/s | 26 kB 00:00 2024-03-29T11:31:32.935 INFO:teuthology.orchestra.run.smithi072.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 484 kB/s | 49 kB 00:00 2024-03-29T11:31:33.036 INFO:teuthology.orchestra.run.smithi072.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 673 kB/s | 67 kB 00:00 2024-03-29T11:31:33.178 INFO:teuthology.orchestra.run.smithi072.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.8 MB/s | 838 kB 00:00 2024-03-29T11:31:33.354 INFO:teuthology.orchestra.run.smithi072.stdout:(27/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 781 kB/s | 9.7 MB 00:12 2024-03-29T11:31:33.379 INFO:teuthology.orchestra.run.smithi072.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 955 kB/s | 192 kB 00:00 2024-03-29T11:31:34.138 INFO:teuthology.orchestra.run.smithi072.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 2.0 MB/s | 1.6 MB 00:00 2024-03-29T11:31:37.378 INFO:teuthology.orchestra.run.smithi089.stdout:(29/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 619 kB/s | 3.4 MB 00:05 2024-03-29T11:31:37.422 INFO:teuthology.orchestra.run.smithi072.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 976 kB/s | 3.1 MB 00:03 2024-03-29T11:31:37.579 INFO:teuthology.orchestra.run.smithi089.stdout:(30/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 670 kB/s | 9.7 MB 00:14 2024-03-29T11:31:38.282 INFO:teuthology.orchestra.run.smithi072.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 705 kB/s | 3.4 MB 00:04 2024-03-29T11:31:41.304 INFO:teuthology.orchestra.run.smithi089.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 816 kB/s | 3.1 MB 00:03 2024-03-29T11:31:44.233 INFO:teuthology.orchestra.run.smithi072.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 861 kB/s | 20 MB 00:23 2024-03-29T11:31:44.236 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:31:44.237 INFO:teuthology.orchestra.run.smithi072.stdout:Total 2.4 MB/s | 58 MB 00:24 2024-03-29T11:31:44.237 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-03-29T11:31:44.274 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-03-29T11:31:44.275 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-03-29T11:31:44.747 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-03-29T11:31:44.747 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-03-29T11:31:46.369 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-03-29T11:31:46.471 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:31:46.514 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:31:46.548 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:31:46.580 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:31:46.596 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:31:46.749 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:31:46.805 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:31:47.124 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:31:47.256 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:31:47.348 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:31:47.462 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:31:47.494 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:31:47.556 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:31:47.594 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:31:47.638 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:31:47.706 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:31:47.740 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:31:47.803 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:31:47.838 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:31:47.947 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:31:47.993 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:31:48.033 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:31:48.067 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:31:48.096 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:31:48.138 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:31:48.457 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:31:48.487 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:31:48.576 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:31:48.609 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:31:48.649 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:31:49.101 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:31:49.134 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:31:49.205 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:31:49.281 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:31:49.314 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:31:51.553 INFO:teuthology.orchestra.run.smithi089.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 706 kB/s | 20 MB 00:28 2024-03-29T11:31:51.557 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:31:51.557 INFO:teuthology.orchestra.run.smithi089.stdout:Total 2.0 MB/s | 58 MB 00:29 2024-03-29T11:31:51.557 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-03-29T11:31:51.595 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-03-29T11:31:51.595 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-03-29T11:31:52.065 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-03-29T11:31:52.066 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-03-29T11:31:53.515 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-03-29T11:31:53.603 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:31:53.685 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:31:53.766 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:31:53.789 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:31:53.806 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:31:53.991 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:31:54.078 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:31:54.403 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:31:54.476 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:31:54.567 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:31:54.655 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:31:54.678 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:31:54.723 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:31:54.755 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:31:54.796 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:31:54.863 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:31:54.888 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:31:54.951 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:31:54.987 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:31:55.098 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:31:55.138 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:31:55.179 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:31:55.211 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:31:55.265 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:31:55.287 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:31:55.306 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:31:55.334 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:31:55.646 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:31:55.669 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:31:55.759 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:31:55.790 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:31:55.832 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:31:55.893 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:31:55.934 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:31:56.015 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:31:56.015 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:31:56.015 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:31:56.015 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:31:56.053 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:31:56.322 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:31:56.347 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:31:56.418 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:31:56.486 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:31:56.516 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:31:58.684 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:31:58.729 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:31:59.270 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:31:59.307 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:31:59.391 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:31:59.391 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:31:59.391 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:31:59.391 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:31:59.425 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:32:06.510 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:32:06.511 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /sys 2024-03-29T11:32:06.511 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /proc 2024-03-29T11:32:06.511 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /mnt 2024-03-29T11:32:06.511 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /var/tmp 2024-03-29T11:32:06.511 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /home 2024-03-29T11:32:06.511 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /root 2024-03-29T11:32:06.511 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /tmp 2024-03-29T11:32:06.511 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:32:07.421 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:32:07.445 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:32:07.445 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:32:07.445 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:32:07.445 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:32:07.445 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:32:07.445 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:32:07.448 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:32:07.544 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:32:07.544 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:32:08.755 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:32:08.756 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /sys 2024-03-29T11:32:08.756 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /proc 2024-03-29T11:32:08.756 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /mnt 2024-03-29T11:32:08.756 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /var/tmp 2024-03-29T11:32:08.756 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /home 2024-03-29T11:32:08.756 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /root 2024-03-29T11:32:08.756 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /tmp 2024-03-29T11:32:08.757 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:32:08.904 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:32:08.904 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:32:08.905 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:32:08.905 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:32:08.905 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:32:08.905 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:32:08.905 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:32:08.905 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:32:08.906 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:32:08.906 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:32:08.906 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:32:08.906 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:32:08.906 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:32:08.906 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:32:08.907 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:32:08.907 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:32:08.907 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:32:08.907 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:32:08.907 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:32:08.907 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:32:08.907 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:32:08.907 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:32:08.908 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:32:08.908 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:32:08.908 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:32:08.908 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:32:08.908 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:32:08.908 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:32:08.908 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:32:08.908 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:32:08.908 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:32:08.908 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:32:08.908 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:32:08.908 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:32:09.673 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:32:09.698 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:32:09.698 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:32:09.698 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:32:09.698 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:32:09.698 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:32:09.699 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:32:09.700 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:32:09.786 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:32:09.786 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:32:10.041 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:32:10.041 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:32:10.041 INFO:teuthology.orchestra.run.smithi072.stdout:Upgraded: 2024-03-29T11:32:10.042 INFO:teuthology.orchestra.run.smithi072.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:10.042 INFO:teuthology.orchestra.run.smithi072.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:10.042 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-03-29T11:32:10.042 INFO:teuthology.orchestra.run.smithi072.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:32:10.042 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:10.042 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:10.042 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:10.042 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:10.042 INFO:teuthology.orchestra.run.smithi072.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:32:10.043 INFO:teuthology.orchestra.run.smithi072.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:32:10.043 INFO:teuthology.orchestra.run.smithi072.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:32:10.043 INFO:teuthology.orchestra.run.smithi072.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:10.043 INFO:teuthology.orchestra.run.smithi072.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:32:10.043 INFO:teuthology.orchestra.run.smithi072.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:32:10.043 INFO:teuthology.orchestra.run.smithi072.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:32:10.043 INFO:teuthology.orchestra.run.smithi072.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:10.043 INFO:teuthology.orchestra.run.smithi072.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:32:10.043 INFO:teuthology.orchestra.run.smithi072.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:10.044 INFO:teuthology.orchestra.run.smithi072.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:32:10.044 INFO:teuthology.orchestra.run.smithi072.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:32:10.044 INFO:teuthology.orchestra.run.smithi072.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:32:10.044 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:10.044 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:10.044 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:10.044 INFO:teuthology.orchestra.run.smithi072.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:32:10.045 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:10.045 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:10.045 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:10.045 INFO:teuthology.orchestra.run.smithi072.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:32:10.045 INFO:teuthology.orchestra.run.smithi072.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:32:10.045 INFO:teuthology.orchestra.run.smithi072.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:32:10.045 INFO:teuthology.orchestra.run.smithi072.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:32:10.045 INFO:teuthology.orchestra.run.smithi072.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:32:10.046 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:32:10.046 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-29T11:32:10.448 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-test 2024-03-29T11:32:10.895 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:32:10.895 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:32:10.895 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:32:10.895 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:32:10.895 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:32:10.895 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:32:10.895 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:32:10.895 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:32:10.895 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:32:10.896 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:32:10.896 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:32:10.896 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:32:10.898 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:32:10.898 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:32:10.898 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:32:10.898 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:32:10.898 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:32:10.898 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:32:10.898 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:32:10.898 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:32:10.898 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:32:10.899 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:32:10.899 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:32:10.899 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:32:10.899 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:32:10.899 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:32:10.899 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:32:10.899 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:32:10.899 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:32:10.899 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:32:10.899 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:32:10.899 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:32:10.899 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:32:10.899 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:32:11.008 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:00:54 ago on Fri 29 Mar 2024 11:31:17 AM UTC. 2024-03-29T11:32:11.164 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-29T11:32:11.165 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-29T11:32:11.165 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2024-03-29T11:32:11.165 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-29T11:32:11.165 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-03-29T11:32:11.165 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:32:11.165 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2024-03-29T11:32:11.165 INFO:teuthology.orchestra.run.smithi072.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:32:11.165 INFO:teuthology.orchestra.run.smithi072.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:32:11.166 INFO:teuthology.orchestra.run.smithi072.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:32:11.166 INFO:teuthology.orchestra.run.smithi072.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:32:11.166 INFO:teuthology.orchestra.run.smithi072.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:32:11.166 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:32:11.166 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-03-29T11:32:11.166 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-29T11:32:11.166 INFO:teuthology.orchestra.run.smithi072.stdout:Install 6 Packages 2024-03-29T11:32:11.166 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:32:11.167 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 48 M 2024-03-29T11:32:11.167 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 204 M 2024-03-29T11:32:11.167 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-03-29T11:32:11.480 INFO:teuthology.orchestra.run.smithi072.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-29T11:32:11.558 INFO:teuthology.orchestra.run.smithi072.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 3.9 MB/s | 300 kB 00:00 2024-03-29T11:32:11.597 INFO:teuthology.orchestra.run.smithi072.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-29T11:32:11.623 INFO:teuthology.orchestra.run.smithi072.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 513 kB/s | 163 kB 00:00 2024-03-29T11:32:11.790 INFO:teuthology.orchestra.run.smithi072.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 973 kB/s | 187 kB 00:00 2024-03-29T11:32:11.895 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:32:11.895 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:32:11.896 INFO:teuthology.orchestra.run.smithi089.stdout:Upgraded: 2024-03-29T11:32:11.896 INFO:teuthology.orchestra.run.smithi089.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:11.896 INFO:teuthology.orchestra.run.smithi089.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:11.896 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-03-29T11:32:11.896 INFO:teuthology.orchestra.run.smithi089.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:32:11.896 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:11.896 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:11.896 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:11.896 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:11.897 INFO:teuthology.orchestra.run.smithi089.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:32:11.897 INFO:teuthology.orchestra.run.smithi089.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:32:11.897 INFO:teuthology.orchestra.run.smithi089.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:32:11.897 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:11.897 INFO:teuthology.orchestra.run.smithi089.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:32:11.897 INFO:teuthology.orchestra.run.smithi089.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:32:11.897 INFO:teuthology.orchestra.run.smithi089.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:32:11.897 INFO:teuthology.orchestra.run.smithi089.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:11.898 INFO:teuthology.orchestra.run.smithi089.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:32:11.898 INFO:teuthology.orchestra.run.smithi089.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:11.898 INFO:teuthology.orchestra.run.smithi089.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:32:11.898 INFO:teuthology.orchestra.run.smithi089.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:32:11.898 INFO:teuthology.orchestra.run.smithi089.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:32:11.898 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:11.898 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:11.898 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:11.899 INFO:teuthology.orchestra.run.smithi089.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:32:11.899 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:11.899 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:11.899 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:11.899 INFO:teuthology.orchestra.run.smithi089.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:32:11.899 INFO:teuthology.orchestra.run.smithi089.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:32:11.899 INFO:teuthology.orchestra.run.smithi089.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:32:11.899 INFO:teuthology.orchestra.run.smithi089.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:32:11.900 INFO:teuthology.orchestra.run.smithi089.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:32:11.900 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:32:11.900 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-29T11:32:12.162 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-test 2024-03-29T11:32:12.716 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:00:53 ago on Fri 29 Mar 2024 11:31:19 AM UTC. 2024-03-29T11:32:12.879 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-29T11:32:12.880 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-29T11:32:12.880 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-03-29T11:32:12.880 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-29T11:32:12.880 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-03-29T11:32:12.880 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:32:12.880 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-03-29T11:32:12.880 INFO:teuthology.orchestra.run.smithi089.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:32:12.881 INFO:teuthology.orchestra.run.smithi089.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:32:12.881 INFO:teuthology.orchestra.run.smithi089.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:32:12.881 INFO:teuthology.orchestra.run.smithi089.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:32:12.881 INFO:teuthology.orchestra.run.smithi089.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:32:12.881 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:32:12.881 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-03-29T11:32:12.881 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-29T11:32:12.881 INFO:teuthology.orchestra.run.smithi089.stdout:Install 6 Packages 2024-03-29T11:32:12.881 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:32:12.882 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 48 M 2024-03-29T11:32:12.882 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 204 M 2024-03-29T11:32:12.882 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-03-29T11:32:13.105 INFO:teuthology.orchestra.run.smithi089.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-03-29T11:32:13.147 INFO:teuthology.orchestra.run.smithi089.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-03-29T11:32:13.180 INFO:teuthology.orchestra.run.smithi089.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-29T11:32:13.314 INFO:teuthology.orchestra.run.smithi089.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-29T11:32:13.931 INFO:teuthology.orchestra.run.smithi089.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 167 kB/s | 163 kB 00:00 2024-03-29T11:32:43.121 INFO:teuthology.orchestra.run.smithi089.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 1.6 MB/s | 47 MB 00:30 2024-03-29T11:32:43.123 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:43.123 INFO:teuthology.orchestra.run.smithi089.stdout:Total 1.6 MB/s | 48 MB 00:30 2024-03-29T11:32:43.135 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-03-29T11:32:43.155 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-03-29T11:32:43.155 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-03-29T11:32:43.532 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-03-29T11:32:43.532 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-03-29T11:32:44.108 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-03-29T11:32:44.187 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:32:44.240 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:32:44.271 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:32:44.312 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:32:44.340 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:32:45.331 INFO:teuthology.orchestra.run.smithi072.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 1.4 MB/s | 47 MB 00:34 2024-03-29T11:32:45.334 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:45.334 INFO:teuthology.orchestra.run.smithi072.stdout:Total 1.4 MB/s | 48 MB 00:34 2024-03-29T11:32:45.347 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-03-29T11:32:45.369 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-03-29T11:32:45.369 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-03-29T11:32:45.752 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-03-29T11:32:45.753 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-03-29T11:32:46.355 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-03-29T11:32:46.425 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:32:46.467 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:32:46.519 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:32:46.560 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:32:46.588 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:32:48.798 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:32:48.875 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:32:51.071 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:32:51.153 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:32:51.313 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:32:51.313 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:32:51.313 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:32:51.313 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:32:51.313 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:32:51.314 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:32:51.670 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:32:52.900 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:32:52.901 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-03-29T11:32:52.901 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:52.901 INFO:teuthology.orchestra.run.smithi089.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:32:52.901 INFO:teuthology.orchestra.run.smithi089.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:52.901 INFO:teuthology.orchestra.run.smithi089.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:32:52.901 INFO:teuthology.orchestra.run.smithi089.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:32:52.901 INFO:teuthology.orchestra.run.smithi089.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:32:52.901 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:32:52.901 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-29T11:32:52.903 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph 2024-03-29T11:32:53.480 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:34 ago on Fri 29 Mar 2024 11:31:19 AM UTC. 2024-03-29T11:32:53.634 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-29T11:32:53.636 INFO:teuthology.orchestra.run.smithi089.stdout:========================================================================================= 2024-03-29T11:32:53.637 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-03-29T11:32:53.637 INFO:teuthology.orchestra.run.smithi089.stdout:========================================================================================= 2024-03-29T11:32:53.637 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-03-29T11:32:53.637 INFO:teuthology.orchestra.run.smithi089.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:32:53.637 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-03-29T11:32:53.637 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:32:53.637 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:32:53.637 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:32:53.637 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:32:53.637 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:32:53.638 INFO:teuthology.orchestra.run.smithi089.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:32:53.638 INFO:teuthology.orchestra.run.smithi089.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:32:53.638 INFO:teuthology.orchestra.run.smithi089.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:32:53.638 INFO:teuthology.orchestra.run.smithi089.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:32:53.638 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:32:53.638 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:32:53.638 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:32:53.638 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:32:53.638 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:32:53.638 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:32:53.638 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:32:53.638 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:32:53.639 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:32:53.639 INFO:teuthology.orchestra.run.smithi089.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:32:53.639 INFO:teuthology.orchestra.run.smithi089.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:32:53.639 INFO:teuthology.orchestra.run.smithi089.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:32:53.639 INFO:teuthology.orchestra.run.smithi089.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:32:53.639 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:32:53.639 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:32:53.639 INFO:teuthology.orchestra.run.smithi089.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:32:53.639 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:32:53.639 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:32:53.639 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:32:53.639 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:32:53.639 INFO:teuthology.orchestra.run.smithi089.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:32:53.639 INFO:teuthology.orchestra.run.smithi089.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:32:53.640 INFO:teuthology.orchestra.run.smithi089.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:32:53.640 INFO:teuthology.orchestra.run.smithi089.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:32:53.640 INFO:teuthology.orchestra.run.smithi089.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:32:53.640 INFO:teuthology.orchestra.run.smithi089.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:32:53.640 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:32:53.640 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-03-29T11:32:53.640 INFO:teuthology.orchestra.run.smithi089.stdout:========================================================================================= 2024-03-29T11:32:53.640 INFO:teuthology.orchestra.run.smithi089.stdout:Install 36 Packages 2024-03-29T11:32:53.640 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:32:53.641 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 30 M 2024-03-29T11:32:53.641 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 107 M 2024-03-29T11:32:53.641 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-03-29T11:32:54.327 INFO:teuthology.orchestra.run.smithi089.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 33 kB/s | 6.6 kB 00:00 2024-03-29T11:32:54.887 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:32:54.888 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:32:54.888 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:32:54.888 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:32:54.888 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:32:54.888 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:32:54.927 INFO:teuthology.orchestra.run.smithi089.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 1.9 MB/s | 1.5 MB 00:00 2024-03-29T11:32:55.069 INFO:teuthology.orchestra.run.smithi089.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 2.3 MB/s | 2.1 MB 00:00 2024-03-29T11:32:55.169 INFO:teuthology.orchestra.run.smithi089.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 2.4 MB/s | 247 kB 00:00 2024-03-29T11:32:55.278 INFO:teuthology.orchestra.run.smithi089.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2024-03-29T11:32:55.320 INFO:teuthology.orchestra.run.smithi089.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T11:32:55.331 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:32:55.331 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:32:55.331 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-03-29T11:32:55.331 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:55.331 INFO:teuthology.orchestra.run.smithi072.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:32:55.332 INFO:teuthology.orchestra.run.smithi072.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:55.332 INFO:teuthology.orchestra.run.smithi072.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:32:55.332 INFO:teuthology.orchestra.run.smithi072.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:32:55.332 INFO:teuthology.orchestra.run.smithi072.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:32:55.332 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:32:55.332 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-29T11:32:55.354 INFO:teuthology.orchestra.run.smithi089.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 943 kB/s | 32 kB 00:00 2024-03-29T11:32:55.387 INFO:teuthology.orchestra.run.smithi089.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T11:32:55.430 INFO:teuthology.orchestra.run.smithi089.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 926 kB/s | 39 kB 00:00 2024-03-29T11:32:55.471 INFO:teuthology.orchestra.run.smithi089.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-29T11:32:55.538 INFO:teuthology.orchestra.run.smithi089.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-29T11:32:55.572 INFO:teuthology.orchestra.run.smithi089.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-29T11:32:55.614 INFO:teuthology.orchestra.run.smithi089.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-29T11:32:55.648 INFO:teuthology.orchestra.run.smithi089.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-29T11:32:55.679 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph 2024-03-29T11:32:56.206 INFO:teuthology.orchestra.run.smithi089.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 224 kB/s | 125 kB 00:00 2024-03-29T11:32:56.241 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 11:31:17 AM UTC. 2024-03-29T11:32:56.307 INFO:teuthology.orchestra.run.smithi089.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 223 kB/s | 22 kB 00:00 2024-03-29T11:32:56.398 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-29T11:32:56.400 INFO:teuthology.orchestra.run.smithi072.stdout:========================================================================================= 2024-03-29T11:32:56.400 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2024-03-29T11:32:56.400 INFO:teuthology.orchestra.run.smithi072.stdout:========================================================================================= 2024-03-29T11:32:56.401 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-03-29T11:32:56.401 INFO:teuthology.orchestra.run.smithi072.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:32:56.401 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2024-03-29T11:32:56.401 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:32:56.401 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:32:56.401 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:32:56.401 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:32:56.401 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:32:56.401 INFO:teuthology.orchestra.run.smithi072.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:32:56.401 INFO:teuthology.orchestra.run.smithi072.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:32:56.401 INFO:teuthology.orchestra.run.smithi072.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:32:56.401 INFO:teuthology.orchestra.run.smithi072.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:32:56.401 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:32:56.402 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:32:56.402 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:32:56.402 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:32:56.402 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:32:56.402 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:32:56.402 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:32:56.402 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:32:56.402 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:32:56.402 INFO:teuthology.orchestra.run.smithi072.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:32:56.402 INFO:teuthology.orchestra.run.smithi072.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:32:56.402 INFO:teuthology.orchestra.run.smithi072.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:32:56.403 INFO:teuthology.orchestra.run.smithi072.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:32:56.403 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:32:56.403 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:32:56.403 INFO:teuthology.orchestra.run.smithi072.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:32:56.403 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:32:56.403 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:32:56.403 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:32:56.403 INFO:teuthology.orchestra.run.smithi072.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:32:56.403 INFO:teuthology.orchestra.run.smithi072.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:32:56.403 INFO:teuthology.orchestra.run.smithi072.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:32:56.403 INFO:teuthology.orchestra.run.smithi072.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:32:56.403 INFO:teuthology.orchestra.run.smithi072.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:32:56.403 INFO:teuthology.orchestra.run.smithi072.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:32:56.403 INFO:teuthology.orchestra.run.smithi072.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:32:56.404 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:32:56.404 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-03-29T11:32:56.404 INFO:teuthology.orchestra.run.smithi072.stdout:========================================================================================= 2024-03-29T11:32:56.404 INFO:teuthology.orchestra.run.smithi072.stdout:Install 36 Packages 2024-03-29T11:32:56.404 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:32:56.404 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 30 M 2024-03-29T11:32:56.404 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 107 M 2024-03-29T11:32:56.404 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-03-29T11:32:56.499 INFO:teuthology.orchestra.run.smithi089.stdout:(17/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 2.2 MB/s | 4.7 MB 00:02 2024-03-29T11:32:56.607 INFO:teuthology.orchestra.run.smithi089.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 503 kB/s | 151 kB 00:00 2024-03-29T11:32:56.866 INFO:teuthology.orchestra.run.smithi089.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 665 kB/s | 172 kB 00:00 2024-03-29T11:32:56.883 INFO:teuthology.orchestra.run.smithi089.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 113 kB/s | 43 kB 00:00 2024-03-29T11:32:56.894 INFO:teuthology.orchestra.run.smithi072.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 38 kB/s | 6.6 kB 00:00 2024-03-29T11:32:56.950 INFO:teuthology.orchestra.run.smithi089.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 160 kB/s | 11 kB 00:00 2024-03-29T11:32:56.983 INFO:teuthology.orchestra.run.smithi089.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.0 MB/s | 358 kB 00:00 2024-03-29T11:32:57.009 INFO:teuthology.orchestra.run.smithi089.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 301 kB/s | 18 kB 00:00 2024-03-29T11:32:57.051 INFO:teuthology.orchestra.run.smithi089.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 345 kB/s | 23 kB 00:00 2024-03-29T11:32:57.076 INFO:teuthology.orchestra.run.smithi089.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 289 kB/s | 19 kB 00:00 2024-03-29T11:32:57.118 INFO:teuthology.orchestra.run.smithi089.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 293 kB/s | 20 kB 00:00 2024-03-29T11:32:57.160 INFO:teuthology.orchestra.run.smithi089.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 553 kB/s | 46 kB 00:00 2024-03-29T11:32:57.185 INFO:teuthology.orchestra.run.smithi089.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.1 MB/s | 79 kB 00:00 2024-03-29T11:32:57.252 INFO:teuthology.orchestra.run.smithi089.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 246 kB/s | 16 kB 00:00 2024-03-29T11:32:57.286 INFO:teuthology.orchestra.run.smithi089.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-03-29T11:32:57.328 INFO:teuthology.orchestra.run.smithi089.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-03-29T11:32:57.370 INFO:teuthology.orchestra.run.smithi089.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 426 kB/s | 36 kB 00:00 2024-03-29T11:32:57.437 INFO:teuthology.orchestra.run.smithi089.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-03-29T11:32:57.496 INFO:teuthology.orchestra.run.smithi089.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.3 MB/s | 427 kB 00:00 2024-03-29T11:32:57.521 INFO:teuthology.orchestra.run.smithi089.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 238 kB/s | 20 kB 00:00 2024-03-29T11:32:57.878 INFO:teuthology.orchestra.run.smithi072.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 1.8 MB/s | 2.1 MB 00:01 2024-03-29T11:32:57.929 INFO:teuthology.orchestra.run.smithi072.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 1.2 MB/s | 1.5 MB 00:01 2024-03-29T11:32:58.313 INFO:teuthology.orchestra.run.smithi089.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 5.0 MB/s | 17 MB 00:03 2024-03-29T11:32:58.317 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:58.317 INFO:teuthology.orchestra.run.smithi089.stdout:Total 6.3 MB/s | 30 MB 00:04 2024-03-29T11:32:58.343 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-03-29T11:32:58.385 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-03-29T11:32:58.385 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-03-29T11:32:58.464 INFO:teuthology.orchestra.run.smithi072.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 462 kB/s | 247 kB 00:00 2024-03-29T11:32:58.581 INFO:teuthology.orchestra.run.smithi072.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-29T11:32:58.623 INFO:teuthology.orchestra.run.smithi072.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T11:32:58.657 INFO:teuthology.orchestra.run.smithi072.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 939 kB/s | 32 kB 00:00 2024-03-29T11:32:58.691 INFO:teuthology.orchestra.run.smithi072.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T11:32:58.733 INFO:teuthology.orchestra.run.smithi072.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 922 kB/s | 39 kB 00:00 2024-03-29T11:32:58.776 INFO:teuthology.orchestra.run.smithi072.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-29T11:32:58.793 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-03-29T11:32:58.794 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-03-29T11:32:58.860 INFO:teuthology.orchestra.run.smithi072.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-29T11:32:58.902 INFO:teuthology.orchestra.run.smithi072.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-03-29T11:32:58.936 INFO:teuthology.orchestra.run.smithi072.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-29T11:32:58.986 INFO:teuthology.orchestra.run.smithi072.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2024-03-29T11:32:59.095 INFO:teuthology.orchestra.run.smithi072.stdout:(15/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 2.1 MB/s | 4.7 MB 00:02 2024-03-29T11:32:59.154 INFO:teuthology.orchestra.run.smithi072.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 746 kB/s | 125 kB 00:00 2024-03-29T11:32:59.347 INFO:teuthology.orchestra.run.smithi072.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 788 kB/s | 151 kB 00:00 2024-03-29T11:32:59.372 INFO:teuthology.orchestra.run.smithi072.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 81 kB/s | 22 kB 00:00 2024-03-29T11:32:59.407 INFO:teuthology.orchestra.run.smithi072.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 716 kB/s | 43 kB 00:00 2024-03-29T11:32:59.500 INFO:teuthology.orchestra.run.smithi072.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.8 MB/s | 358 kB 00:00 2024-03-29T11:32:59.534 INFO:teuthology.orchestra.run.smithi072.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 318 kB/s | 11 kB 00:00 2024-03-29T11:32:59.559 INFO:teuthology.orchestra.run.smithi072.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 920 kB/s | 172 kB 00:00 2024-03-29T11:32:59.585 INFO:teuthology.orchestra.run.smithi072.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 347 kB/s | 18 kB 00:00 2024-03-29T11:32:59.619 INFO:teuthology.orchestra.run.smithi072.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 392 kB/s | 23 kB 00:00 2024-03-29T11:32:59.652 INFO:teuthology.orchestra.run.smithi072.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 288 kB/s | 19 kB 00:00 2024-03-29T11:32:59.678 INFO:teuthology.orchestra.run.smithi072.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 334 kB/s | 20 kB 00:00 2024-03-29T11:32:59.712 INFO:teuthology.orchestra.run.smithi072.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 786 kB/s | 46 kB 00:00 2024-03-29T11:32:59.746 INFO:teuthology.orchestra.run.smithi072.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.1 MB/s | 79 kB 00:00 2024-03-29T11:32:59.788 INFO:teuthology.orchestra.run.smithi072.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-03-29T11:32:59.814 INFO:teuthology.orchestra.run.smithi072.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 243 kB/s | 16 kB 00:00 2024-03-29T11:32:59.839 INFO:teuthology.orchestra.run.smithi072.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-03-29T11:32:59.864 INFO:teuthology.orchestra.run.smithi072.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 707 kB/s | 36 kB 00:00 2024-03-29T11:32:59.890 INFO:teuthology.orchestra.run.smithi072.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 4.4 MB/s | 230 kB 00:00 2024-03-29T11:32:59.926 INFO:teuthology.orchestra.run.smithi072.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 568 kB/s | 20 kB 00:00 2024-03-29T11:32:59.959 INFO:teuthology.orchestra.run.smithi072.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.4 MB/s | 427 kB 00:00 2024-03-29T11:33:00.117 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-03-29T11:33:00.191 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:33:00.224 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:33:00.251 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:33:00.323 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:33:00.359 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:33:00.392 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:33:00.482 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:33:00.561 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:33:00.603 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:33:00.674 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:33:00.737 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:33:00.776 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:33:00.824 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:33:00.880 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:33:01.062 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:33:01.113 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:33:01.144 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:33:01.179 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:33:01.209 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:33:01.286 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:33:01.319 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:33:01.353 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:33:01.393 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:33:01.445 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:33:01.471 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:33:01.560 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:33:01.602 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:33:01.674 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:33:01.811 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:33:01.945 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:33:02.105 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:33:02.133 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:33:02.159 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:33:02.159 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:02.159 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:33:02.159 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:33:02.159 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:33:02.159 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:03.601 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:33:03.628 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:33:03.628 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:03.628 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:33:03.628 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:33:03.628 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:33:03.628 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:04.077 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:33:04.099 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:33:04.100 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:04.100 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:33:04.100 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:33:04.100 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:33:04.100 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:04.303 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:33:04.326 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:33:04.326 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:04.326 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:33:04.326 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:33:04.326 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:33:04.326 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:04.392 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:33:06.158 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:33:06.158 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:33:06.158 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:33:06.158 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:33:06.158 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:33:06.158 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:33:06.158 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:33:06.158 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:33:06.158 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:33:06.158 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:33:06.159 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:33:06.159 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:33:06.159 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:33:06.159 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:33:06.159 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:33:06.159 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:33:06.159 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:33:06.159 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:33:06.159 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:33:06.159 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:33:06.159 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:33:06.159 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:33:06.159 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:33:06.159 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:33:06.159 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:33:06.160 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:33:06.160 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:33:06.160 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:33:06.161 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:33:06.161 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:33:06.161 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:33:06.161 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:33:06.161 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:33:06.161 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:33:06.161 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:33:06.161 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:33:06.982 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:33:06.983 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:06.983 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-03-29T11:33:06.983 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:06.983 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:06.983 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:06.983 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:06.983 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:06.983 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:06.983 INFO:teuthology.orchestra.run.smithi089.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:33:06.983 INFO:teuthology.orchestra.run.smithi089.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:33:06.983 INFO:teuthology.orchestra.run.smithi089.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:33:06.983 INFO:teuthology.orchestra.run.smithi089.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:33:06.983 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:33:06.983 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:33:06.984 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:33:06.984 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:33:06.984 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:33:06.984 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:33:06.984 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:33:06.984 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:33:06.984 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:33:06.984 INFO:teuthology.orchestra.run.smithi089.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:33:06.984 INFO:teuthology.orchestra.run.smithi089.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:33:06.984 INFO:teuthology.orchestra.run.smithi089.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:33:06.984 INFO:teuthology.orchestra.run.smithi089.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:33:06.984 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:33:06.984 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:33:06.984 INFO:teuthology.orchestra.run.smithi089.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:33:06.985 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:33:06.985 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:33:06.985 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:33:06.985 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:33:06.985 INFO:teuthology.orchestra.run.smithi089.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:33:06.985 INFO:teuthology.orchestra.run.smithi089.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:33:06.985 INFO:teuthology.orchestra.run.smithi089.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:33:06.985 INFO:teuthology.orchestra.run.smithi089.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:33:06.985 INFO:teuthology.orchestra.run.smithi089.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:33:06.985 INFO:teuthology.orchestra.run.smithi089.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:33:06.985 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:06.985 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-29T11:33:07.361 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-base 2024-03-29T11:33:07.501 INFO:teuthology.orchestra.run.smithi072.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 1.7 MB/s | 17 MB 00:09 2024-03-29T11:33:07.505 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:07.505 INFO:teuthology.orchestra.run.smithi072.stdout:Total 2.7 MB/s | 30 MB 00:11 2024-03-29T11:33:07.530 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-03-29T11:33:07.572 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-03-29T11:33:07.573 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-03-29T11:33:07.941 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:48 ago on Fri 29 Mar 2024 11:31:19 AM UTC. 2024-03-29T11:33:08.017 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-03-29T11:33:08.017 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-03-29T11:33:08.038 INFO:teuthology.orchestra.run.smithi089.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:08.090 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-29T11:33:08.091 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-03-29T11:33:08.091 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-29T11:33:08.168 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install cephadm 2024-03-29T11:33:08.729 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:49 ago on Fri 29 Mar 2024 11:31:19 AM UTC. 2024-03-29T11:33:08.879 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-29T11:33:08.880 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-29T11:33:08.880 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-03-29T11:33:08.880 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-29T11:33:08.881 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-03-29T11:33:08.881 INFO:teuthology.orchestra.run.smithi089.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:33:08.881 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:08.881 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-03-29T11:33:08.881 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-29T11:33:08.881 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-03-29T11:33:08.881 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:08.881 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 747 k 2024-03-29T11:33:08.881 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 754 k 2024-03-29T11:33:08.881 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-03-29T11:33:09.345 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-03-29T11:33:09.413 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:33:09.474 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:33:09.516 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:33:09.596 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:33:09.643 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:33:09.677 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:33:09.777 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:33:09.856 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:33:09.906 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:33:09.986 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:33:10.043 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:33:10.090 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:33:10.148 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:33:10.212 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:33:10.407 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:33:10.466 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:33:10.497 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:33:10.541 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:33:10.571 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:33:10.648 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:33:10.691 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:33:10.725 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:33:10.767 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:33:10.811 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:33:10.820 INFO:teuthology.orchestra.run.smithi089.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 385 kB/s | 747 kB 00:01 2024-03-29T11:33:10.821 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:10.822 INFO:teuthology.orchestra.run.smithi089.stdout:Total 385 kB/s | 747 kB 00:01 2024-03-29T11:33:10.822 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-03-29T11:33:10.828 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-03-29T11:33:10.828 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-03-29T11:33:10.834 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-03-29T11:33:10.834 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-03-29T11:33:10.846 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:33:10.909 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-03-29T11:33:10.927 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:33:10.970 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:33:11.043 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:33:11.189 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:33:11.319 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:33:11.383 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:33:11.476 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:33:11.477 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:33:11.509 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:33:11.533 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:33:11.533 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:11.533 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:33:11.533 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:33:11.533 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:33:11.533 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:33:11.850 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:33:12.146 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:33:12.147 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:12.147 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-03-29T11:33:12.147 INFO:teuthology.orchestra.run.smithi089.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:12.147 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:12.147 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-29T11:33:12.343 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:33:12.915 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:53 ago on Fri 29 Mar 2024 11:31:19 AM UTC. 2024-03-29T11:33:12.977 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:33:13.004 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:33:13.004 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:13.004 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:33:13.004 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:33:13.004 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:33:13.004 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:33:13.069 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-29T11:33:13.070 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-29T11:33:13.070 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repo Size 2024-03-29T11:33:13.070 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-29T11:33:13.070 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-03-29T11:33:13.070 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:33:13.070 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:13.071 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-03-29T11:33:13.071 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-29T11:33:13.071 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-03-29T11:33:13.071 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:13.071 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 145 k 2024-03-29T11:33:13.071 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 436 k 2024-03-29T11:33:13.071 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-03-29T11:33:13.312 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 603 kB/s | 145 kB 00:00 2024-03-29T11:33:13.312 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:13.312 INFO:teuthology.orchestra.run.smithi089.stdout:Total 599 kB/s | 145 kB 00:00 2024-03-29T11:33:13.313 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-03-29T11:33:13.321 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-03-29T11:33:13.321 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-03-29T11:33:13.373 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-03-29T11:33:13.374 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-03-29T11:33:13.445 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:33:13.469 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:33:13.469 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:13.469 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:33:13.469 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:33:13.469 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:33:13.469 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:33:13.513 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-03-29T11:33:13.571 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:33:13.596 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:33:13.596 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:13.596 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:33:13.596 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:13.680 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:33:13.703 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:33:13.703 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:13.703 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:33:13.703 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:33:13.703 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:33:13.704 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:33:13.762 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:33:14.628 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:33:14.628 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:14.628 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-03-29T11:33:14.628 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:14.628 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:14.628 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-29T11:33:14.867 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-mgr 2024-03-29T11:33:15.438 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:56 ago on Fri 29 Mar 2024 11:31:19 AM UTC. 2024-03-29T11:33:15.535 INFO:teuthology.orchestra.run.smithi089.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:15.587 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-29T11:33:15.588 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-03-29T11:33:15.588 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-29T11:33:15.665 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:33:16.226 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:57 ago on Fri 29 Mar 2024 11:31:19 AM UTC. 2024-03-29T11:33:16.276 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:33:16.276 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:33:16.276 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:33:16.276 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:33:16.276 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:33:16.276 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:33:16.276 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:33:16.276 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:33:16.276 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:33:16.277 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:33:16.277 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:33:16.277 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:33:16.277 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:33:16.277 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:33:16.277 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:33:16.277 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:33:16.277 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:33:16.277 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:33:16.277 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:33:16.277 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:33:16.277 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:33:16.277 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:33:16.277 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:33:16.277 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:33:16.277 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:33:16.278 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:33:16.278 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:33:16.278 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:33:16.279 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:33:16.279 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:33:16.279 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:33:16.279 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:33:16.279 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:33:16.279 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:33:16.279 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:33:16.279 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:33:16.378 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-29T11:33:16.379 INFO:teuthology.orchestra.run.smithi089.stdout:====================================================================================== 2024-03-29T11:33:16.379 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-03-29T11:33:16.379 INFO:teuthology.orchestra.run.smithi089.stdout:====================================================================================== 2024-03-29T11:33:16.379 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-03-29T11:33:16.380 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:33:16.380 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-03-29T11:33:16.380 INFO:teuthology.orchestra.run.smithi089.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:33:16.380 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:33:16.380 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:33:16.380 INFO:teuthology.orchestra.run.smithi089.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:33:16.380 INFO:teuthology.orchestra.run.smithi089.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:33:16.380 INFO:teuthology.orchestra.run.smithi089.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:33:16.380 INFO:teuthology.orchestra.run.smithi089.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:33:16.380 INFO:teuthology.orchestra.run.smithi089.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:33:16.380 INFO:teuthology.orchestra.run.smithi089.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:33:16.380 INFO:teuthology.orchestra.run.smithi089.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:33:16.380 INFO:teuthology.orchestra.run.smithi089.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:33:16.380 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:16.380 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-03-29T11:33:16.381 INFO:teuthology.orchestra.run.smithi089.stdout:====================================================================================== 2024-03-29T11:33:16.381 INFO:teuthology.orchestra.run.smithi089.stdout:Install 12 Packages 2024-03-29T11:33:16.381 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:16.381 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 6.9 M 2024-03-29T11:33:16.381 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 26 M 2024-03-29T11:33:16.381 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-03-29T11:33:16.651 INFO:teuthology.orchestra.run.smithi089.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 177 kB/s | 31 kB 00:00 2024-03-29T11:33:16.677 INFO:teuthology.orchestra.run.smithi089.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 76 kB/s | 15 kB 00:00 2024-03-29T11:33:16.810 INFO:teuthology.orchestra.run.smithi089.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.7 MB/s | 238 kB 00:00 2024-03-29T11:33:16.861 INFO:teuthology.orchestra.run.smithi089.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.8 MB/s | 1.0 MB 00:00 2024-03-29T11:33:16.919 INFO:teuthology.orchestra.run.smithi089.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 4.0 MB/s | 1.8 MB 00:00 2024-03-29T11:33:17.078 INFO:teuthology.orchestra.run.smithi089.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.2 MB/s | 863 kB 00:00 2024-03-29T11:33:17.103 INFO:teuthology.orchestra.run.smithi089.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 106 kB/s | 19 kB 00:00 2024-03-29T11:33:17.237 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:33:17.238 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:33:17.238 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-03-29T11:33:17.238 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:17.238 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:17.238 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:17.238 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:17.238 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:17.238 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:17.238 INFO:teuthology.orchestra.run.smithi072.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:33:17.238 INFO:teuthology.orchestra.run.smithi072.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:33:17.238 INFO:teuthology.orchestra.run.smithi072.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:33:17.238 INFO:teuthology.orchestra.run.smithi072.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:33:17.238 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:33:17.239 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:33:17.239 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:33:17.239 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:33:17.239 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:33:17.239 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:33:17.239 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:33:17.239 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:33:17.239 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:33:17.239 INFO:teuthology.orchestra.run.smithi072.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:33:17.239 INFO:teuthology.orchestra.run.smithi072.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:33:17.239 INFO:teuthology.orchestra.run.smithi072.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:33:17.239 INFO:teuthology.orchestra.run.smithi072.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:33:17.239 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:33:17.239 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:33:17.239 INFO:teuthology.orchestra.run.smithi072.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:33:17.239 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:33:17.240 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:33:17.240 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:33:17.240 INFO:teuthology.orchestra.run.smithi072.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:33:17.240 INFO:teuthology.orchestra.run.smithi072.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:33:17.240 INFO:teuthology.orchestra.run.smithi072.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:33:17.240 INFO:teuthology.orchestra.run.smithi072.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:33:17.240 INFO:teuthology.orchestra.run.smithi072.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:33:17.240 INFO:teuthology.orchestra.run.smithi072.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:33:17.240 INFO:teuthology.orchestra.run.smithi072.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:33:17.240 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:33:17.240 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-29T11:33:17.253 INFO:teuthology.orchestra.run.smithi089.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.4 MB/s | 548 kB 00:00 2024-03-29T11:33:17.320 INFO:teuthology.orchestra.run.smithi089.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 461 kB/s | 31 kB 00:00 2024-03-29T11:33:17.362 INFO:teuthology.orchestra.run.smithi089.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 558 kB/s | 144 kB 00:00 2024-03-29T11:33:17.387 INFO:teuthology.orchestra.run.smithi089.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-03-29T11:33:17.496 INFO:teuthology.orchestra.run.smithi089.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 4.9 MB/s | 2.0 MB 00:00 2024-03-29T11:33:17.499 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:17.499 INFO:teuthology.orchestra.run.smithi089.stdout:Total 6.1 MB/s | 6.9 MB 00:01 2024-03-29T11:33:17.518 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-03-29T11:33:17.535 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-03-29T11:33:17.535 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-03-29T11:33:17.678 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-base 2024-03-29T11:33:17.715 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-03-29T11:33:17.715 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-03-29T11:33:18.247 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:01 ago on Fri 29 Mar 2024 11:31:17 AM UTC. 2024-03-29T11:33:18.260 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-03-29T11:33:18.338 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:33:18.345 INFO:teuthology.orchestra.run.smithi072.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:18.374 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:33:18.397 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-29T11:33:18.398 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2024-03-29T11:33:18.398 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-29T11:33:18.420 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:33:18.468 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:33:18.476 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install cephadm 2024-03-29T11:33:18.568 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:33:18.692 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:33:18.796 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:33:18.839 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:33:18.868 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:33:18.900 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:33:19.036 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:02 ago on Fri 29 Mar 2024 11:31:17 AM UTC. 2024-03-29T11:33:19.143 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:33:19.175 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:33:19.187 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-29T11:33:19.188 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-29T11:33:19.188 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2024-03-29T11:33:19.188 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-29T11:33:19.188 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-03-29T11:33:19.188 INFO:teuthology.orchestra.run.smithi072.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:33:19.189 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:33:19.189 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-03-29T11:33:19.189 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-29T11:33:19.189 INFO:teuthology.orchestra.run.smithi072.stdout:Install 1 Package 2024-03-29T11:33:19.189 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:33:19.189 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 747 k 2024-03-29T11:33:19.189 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 754 k 2024-03-29T11:33:19.190 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-03-29T11:33:19.469 INFO:teuthology.orchestra.run.smithi072.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 2.6 MB/s | 747 kB 00:00 2024-03-29T11:33:19.470 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:19.470 INFO:teuthology.orchestra.run.smithi072.stdout:Total 2.6 MB/s | 747 kB 00:00 2024-03-29T11:33:19.470 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-03-29T11:33:19.475 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-03-29T11:33:19.475 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-03-29T11:33:19.481 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-03-29T11:33:19.481 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-03-29T11:33:19.556 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-03-29T11:33:20.004 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:33:20.018 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:33:20.019 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:33:20.019 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:33:20.019 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:33:20.019 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:33:20.019 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:33:20.019 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:33:20.019 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:33:20.019 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:33:20.020 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:33:20.020 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:33:20.020 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:33:20.104 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:33:20.371 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:33:20.371 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:20.371 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-03-29T11:33:20.371 INFO:teuthology.orchestra.run.smithi089.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:33:20.371 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:20.371 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:20.371 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:20.372 INFO:teuthology.orchestra.run.smithi089.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:33:20.372 INFO:teuthology.orchestra.run.smithi089.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:33:20.372 INFO:teuthology.orchestra.run.smithi089.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:33:20.372 INFO:teuthology.orchestra.run.smithi089.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:33:20.372 INFO:teuthology.orchestra.run.smithi089.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:33:20.372 INFO:teuthology.orchestra.run.smithi089.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:33:20.372 INFO:teuthology.orchestra.run.smithi089.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:33:20.372 INFO:teuthology.orchestra.run.smithi089.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:33:20.372 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:20.372 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-29T11:33:20.534 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:33:20.640 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:33:20.827 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:33:20.828 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:33:20.828 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-03-29T11:33:20.828 INFO:teuthology.orchestra.run.smithi072.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:20.828 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:33:20.828 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-29T11:33:21.055 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:33:21.203 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:02:02 ago on Fri 29 Mar 2024 11:31:19 AM UTC. 2024-03-29T11:33:21.355 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-29T11:33:21.356 INFO:teuthology.orchestra.run.smithi089.stdout:============================================================================================ 2024-03-29T11:33:21.356 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-03-29T11:33:21.356 INFO:teuthology.orchestra.run.smithi089.stdout:============================================================================================ 2024-03-29T11:33:21.356 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-03-29T11:33:21.356 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:33:21.356 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-03-29T11:33:21.356 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:33:21.356 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:33:21.356 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:33:21.356 INFO:teuthology.orchestra.run.smithi089.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:33:21.356 INFO:teuthology.orchestra.run.smithi089.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:33:21.357 INFO:teuthology.orchestra.run.smithi089.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:33:21.357 INFO:teuthology.orchestra.run.smithi089.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:33:21.357 INFO:teuthology.orchestra.run.smithi089.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:33:21.357 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:33:21.357 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:33:21.357 INFO:teuthology.orchestra.run.smithi089.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:33:21.357 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:21.357 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-03-29T11:33:21.357 INFO:teuthology.orchestra.run.smithi089.stdout:============================================================================================ 2024-03-29T11:33:21.357 INFO:teuthology.orchestra.run.smithi089.stdout:Install 12 Packages 2024-03-29T11:33:21.357 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:21.358 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 37 M 2024-03-29T11:33:21.358 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 209 M 2024-03-29T11:33:21.358 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-03-29T11:33:21.626 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:04 ago on Fri 29 Mar 2024 11:31:17 AM UTC. 2024-03-29T11:33:21.664 INFO:teuthology.orchestra.run.smithi089.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 238 kB/s | 32 kB 00:00 2024-03-29T11:33:21.714 INFO:teuthology.orchestra.run.smithi089.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 351 kB/s | 17 kB 00:00 2024-03-29T11:33:21.765 INFO:teuthology.orchestra.run.smithi089.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 13 MB/s | 3.0 MB 00:00 2024-03-29T11:33:21.782 INFO:teuthology.orchestra.run.smithi089.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 525 kB/s | 34 kB 00:00 2024-03-29T11:33:21.783 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-29T11:33:21.783 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-29T11:33:21.783 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repo Size 2024-03-29T11:33:21.784 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-29T11:33:21.784 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-03-29T11:33:21.784 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:33:21.784 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:33:21.784 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-03-29T11:33:21.784 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-29T11:33:21.784 INFO:teuthology.orchestra.run.smithi072.stdout:Install 1 Package 2024-03-29T11:33:21.784 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:33:21.784 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 145 k 2024-03-29T11:33:21.785 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 436 k 2024-03-29T11:33:21.785 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-03-29T11:33:21.957 INFO:teuthology.orchestra.run.smithi089.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 25 MB/s | 4.7 MB 00:00 2024-03-29T11:33:22.024 INFO:teuthology.orchestra.run.smithi089.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-03-29T11:33:22.049 INFO:teuthology.orchestra.run.smithi089.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-29T11:33:22.220 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 334 kB/s | 145 kB 00:00 2024-03-29T11:33:22.220 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:22.221 INFO:teuthology.orchestra.run.smithi072.stdout:Total 333 kB/s | 145 kB 00:00 2024-03-29T11:33:22.221 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-03-29T11:33:22.228 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-03-29T11:33:22.229 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-03-29T11:33:22.280 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-03-29T11:33:22.281 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-03-29T11:33:22.418 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-03-29T11:33:22.425 INFO:teuthology.orchestra.run.smithi089.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2024-03-29T11:33:22.514 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:33:22.538 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:33:22.538 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:22.539 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:33:22.539 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:33:22.550 INFO:teuthology.orchestra.run.smithi089.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 803 kB 00:00 2024-03-29T11:33:22.634 INFO:teuthology.orchestra.run.smithi089.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 938 kB/s | 195 kB 00:00 2024-03-29T11:33:23.125 INFO:teuthology.orchestra.run.smithi089.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-1 4.6 MB/s | 7.4 MB 00:01 2024-03-29T11:33:23.151 INFO:teuthology.orchestra.run.smithi089.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 409 kB/s | 245 kB 00:00 2024-03-29T11:33:23.154 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:23.154 INFO:teuthology.orchestra.run.smithi089.stdout:Total 21 MB/s | 37 MB 00:01 2024-03-29T11:33:23.194 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-03-29T11:33:23.218 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-03-29T11:33:23.218 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-03-29T11:33:23.543 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:33:23.543 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:33:23.543 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-03-29T11:33:23.544 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:23.544 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:33:23.544 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-29T11:33:23.597 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-03-29T11:33:23.597 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-03-29T11:33:23.829 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-mgr 2024-03-29T11:33:24.288 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-03-29T11:33:24.357 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:33:24.389 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:07 ago on Fri 29 Mar 2024 11:31:17 AM UTC. 2024-03-29T11:33:24.397 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:33:24.490 INFO:teuthology.orchestra.run.smithi072.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:24.490 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:33:24.533 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:33:24.540 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-29T11:33:24.542 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2024-03-29T11:33:24.542 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-29T11:33:24.617 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:33:24.710 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:33:24.749 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:33:24.831 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:33:25.170 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:08 ago on Fri 29 Mar 2024 11:31:17 AM UTC. 2024-03-29T11:33:25.320 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-29T11:33:25.321 INFO:teuthology.orchestra.run.smithi072.stdout:====================================================================================== 2024-03-29T11:33:25.321 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2024-03-29T11:33:25.321 INFO:teuthology.orchestra.run.smithi072.stdout:====================================================================================== 2024-03-29T11:33:25.321 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-03-29T11:33:25.321 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:33:25.321 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2024-03-29T11:33:25.321 INFO:teuthology.orchestra.run.smithi072.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:33:25.321 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:33:25.321 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:33:25.322 INFO:teuthology.orchestra.run.smithi072.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:33:25.322 INFO:teuthology.orchestra.run.smithi072.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:33:25.322 INFO:teuthology.orchestra.run.smithi072.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:33:25.322 INFO:teuthology.orchestra.run.smithi072.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:33:25.322 INFO:teuthology.orchestra.run.smithi072.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:33:25.322 INFO:teuthology.orchestra.run.smithi072.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:33:25.322 INFO:teuthology.orchestra.run.smithi072.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:33:25.322 INFO:teuthology.orchestra.run.smithi072.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:33:25.322 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:33:25.322 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-03-29T11:33:25.322 INFO:teuthology.orchestra.run.smithi072.stdout:====================================================================================== 2024-03-29T11:33:25.322 INFO:teuthology.orchestra.run.smithi072.stdout:Install 12 Packages 2024-03-29T11:33:25.322 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:33:25.323 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 6.9 M 2024-03-29T11:33:25.323 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 26 M 2024-03-29T11:33:25.323 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-03-29T11:33:25.481 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:33:25.587 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:33:25.967 INFO:teuthology.orchestra.run.smithi072.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 154 kB/s | 31 kB 00:00 2024-03-29T11:33:25.995 INFO:teuthology.orchestra.run.smithi072.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 67 kB/s | 15 kB 00:00 2024-03-29T11:33:26.150 INFO:teuthology.orchestra.run.smithi072.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.5 MB/s | 238 kB 00:00 2024-03-29T11:33:26.234 INFO:teuthology.orchestra.run.smithi072.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.8 MB/s | 1.0 MB 00:00 2024-03-29T11:33:26.718 INFO:teuthology.orchestra.run.smithi072.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.1 MB/s | 548 kB 00:00 2024-03-29T11:33:26.819 INFO:teuthology.orchestra.run.smithi072.stdout:(6/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 1.7 MB/s | 1.8 MB 00:01 2024-03-29T11:33:26.853 INFO:teuthology.orchestra.run.smithi072.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 145 kB/s | 19 kB 00:00 2024-03-29T11:33:26.862 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:33:26.896 INFO:teuthology.orchestra.run.smithi072.stdout:(8/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.1 MB/s | 863 kB 00:00 2024-03-29T11:33:26.947 INFO:teuthology.orchestra.run.smithi072.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.5 MB/s | 144 kB 00:00 2024-03-29T11:33:27.032 INFO:teuthology.orchestra.run.smithi072.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-03-29T11:33:27.173 INFO:teuthology.orchestra.run.smithi072.stdout:(11/12): python3-repoze-lru-0.7-16.el9.noarch.r 122 kB/s | 31 kB 00:00 2024-03-29T11:33:27.457 INFO:teuthology.orchestra.run.smithi072.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.2 MB/s | 2.0 MB 00:00 2024-03-29T11:33:27.461 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:27.462 INFO:teuthology.orchestra.run.smithi072.stdout:Total 3.2 MB/s | 6.9 MB 00:02 2024-03-29T11:33:27.479 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-03-29T11:33:27.495 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-03-29T11:33:27.495 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-03-29T11:33:27.671 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-03-29T11:33:27.672 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-03-29T11:33:27.904 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:33:27.927 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:33:28.186 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-03-29T11:33:28.266 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:33:28.302 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:33:28.357 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:33:28.389 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:33:28.521 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:33:28.660 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:33:28.756 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:33:28.810 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:33:28.839 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:33:28.870 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:33:29.120 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:33:29.155 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:33:30.253 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:33:30.253 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:33:30.253 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:33:30.253 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:33:30.254 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:33:30.254 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:33:30.254 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:33:30.254 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:33:30.254 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:33:30.254 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:33:30.254 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:33:30.254 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:33:30.579 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:33:30.579 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:33:30.579 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:33:30.579 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:33:30.579 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:33:30.580 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:33:30.580 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:33:30.580 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:33:30.580 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:33:30.580 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:33:30.580 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:33:30.580 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:33:30.596 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:33:30.596 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:33:30.596 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-03-29T11:33:30.596 INFO:teuthology.orchestra.run.smithi072.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:33:30.596 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:30.596 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:30.596 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:30.596 INFO:teuthology.orchestra.run.smithi072.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:33:30.596 INFO:teuthology.orchestra.run.smithi072.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:33:30.596 INFO:teuthology.orchestra.run.smithi072.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:33:30.597 INFO:teuthology.orchestra.run.smithi072.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:33:30.597 INFO:teuthology.orchestra.run.smithi072.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:33:30.597 INFO:teuthology.orchestra.run.smithi072.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:33:30.597 INFO:teuthology.orchestra.run.smithi072.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:33:30.597 INFO:teuthology.orchestra.run.smithi072.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:33:30.597 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:33:30.597 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-29T11:33:30.893 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:33:31.029 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:33:31.029 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:31.029 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-03-29T11:33:31.029 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:31.029 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:33:31.029 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:33:31.029 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:33:31.029 INFO:teuthology.orchestra.run.smithi089.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:33:31.030 INFO:teuthology.orchestra.run.smithi089.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:33:31.030 INFO:teuthology.orchestra.run.smithi089.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:33:31.030 INFO:teuthology.orchestra.run.smithi089.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:33:31.030 INFO:teuthology.orchestra.run.smithi089.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:33:31.030 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:33:31.030 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:33:31.030 INFO:teuthology.orchestra.run.smithi089.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:33:31.030 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:31.030 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-29T11:33:31.380 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:33:31.457 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:14 ago on Fri 29 Mar 2024 11:31:17 AM UTC. 2024-03-29T11:33:31.612 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-29T11:33:31.613 INFO:teuthology.orchestra.run.smithi072.stdout:============================================================================================ 2024-03-29T11:33:31.613 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2024-03-29T11:33:31.613 INFO:teuthology.orchestra.run.smithi072.stdout:============================================================================================ 2024-03-29T11:33:31.613 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-03-29T11:33:31.614 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:33:31.614 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2024-03-29T11:33:31.614 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:33:31.614 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:33:31.614 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:33:31.614 INFO:teuthology.orchestra.run.smithi072.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:33:31.614 INFO:teuthology.orchestra.run.smithi072.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:33:31.614 INFO:teuthology.orchestra.run.smithi072.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:33:31.614 INFO:teuthology.orchestra.run.smithi072.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:33:31.614 INFO:teuthology.orchestra.run.smithi072.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:33:31.614 INFO:teuthology.orchestra.run.smithi072.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:33:31.614 INFO:teuthology.orchestra.run.smithi072.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:33:31.614 INFO:teuthology.orchestra.run.smithi072.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:33:31.614 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:33:31.615 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-03-29T11:33:31.615 INFO:teuthology.orchestra.run.smithi072.stdout:============================================================================================ 2024-03-29T11:33:31.615 INFO:teuthology.orchestra.run.smithi072.stdout:Install 12 Packages 2024-03-29T11:33:31.615 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:33:31.615 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 37 M 2024-03-29T11:33:31.615 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 209 M 2024-03-29T11:33:31.615 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-03-29T11:33:31.956 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:02:12 ago on Fri 29 Mar 2024 11:31:19 AM UTC. 2024-03-29T11:33:32.073 INFO:teuthology.orchestra.run.smithi072.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 248 kB/s | 32 kB 00:00 2024-03-29T11:33:32.107 INFO:teuthology.orchestra.run.smithi072.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 542 kB/s | 17 kB 00:00 2024-03-29T11:33:32.108 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-29T11:33:32.109 INFO:teuthology.orchestra.run.smithi089.stdout:======================================================================================== 2024-03-29T11:33:32.109 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-03-29T11:33:32.109 INFO:teuthology.orchestra.run.smithi089.stdout:======================================================================================== 2024-03-29T11:33:32.109 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-03-29T11:33:32.109 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:33:32.109 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-03-29T11:33:32.109 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:33:32.109 INFO:teuthology.orchestra.run.smithi089.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:33:32.110 INFO:teuthology.orchestra.run.smithi089.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:33:32.110 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:33:32.110 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:33:32.110 INFO:teuthology.orchestra.run.smithi089.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:33:32.110 INFO:teuthology.orchestra.run.smithi089.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:33:32.110 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:33:32.110 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:33:32.110 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:33:32.110 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:33:32.110 INFO:teuthology.orchestra.run.smithi089.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:33:32.110 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:32.110 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-03-29T11:33:32.110 INFO:teuthology.orchestra.run.smithi089.stdout:======================================================================================== 2024-03-29T11:33:32.110 INFO:teuthology.orchestra.run.smithi089.stdout:Install 13 Packages 2024-03-29T11:33:32.110 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:32.111 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 2.0 M 2024-03-29T11:33:32.111 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 26 M 2024-03-29T11:33:32.111 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-03-29T11:33:32.149 INFO:teuthology.orchestra.run.smithi072.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 835 kB/s | 34 kB 00:00 2024-03-29T11:33:32.225 INFO:teuthology.orchestra.run.smithi072.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-29T11:33:32.303 INFO:teuthology.orchestra.run.smithi089.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 171 kB/s | 24 kB 00:00 2024-03-29T11:33:32.328 INFO:teuthology.orchestra.run.smithi089.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 108 kB/s | 18 kB 00:00 2024-03-29T11:33:32.420 INFO:teuthology.orchestra.run.smithi089.stdout:(3/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 201 kB/s | 52 kB 00:00 2024-03-29T11:33:32.446 INFO:teuthology.orchestra.run.smithi089.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.2 MB/s | 174 kB 00:00 2024-03-29T11:33:32.452 INFO:teuthology.orchestra.run.smithi072.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 16 MB/s | 4.7 MB 00:00 2024-03-29T11:33:32.471 INFO:teuthology.orchestra.run.smithi089.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 931 kB/s | 132 kB 00:00 2024-03-29T11:33:32.496 INFO:teuthology.orchestra.run.smithi089.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 211 kB 00:00 2024-03-29T11:33:32.521 INFO:teuthology.orchestra.run.smithi089.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 854 kB/s | 43 kB 00:00 2024-03-29T11:33:32.553 INFO:teuthology.orchestra.run.smithi072.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 15 MB/s | 4.8 MB 00:00 2024-03-29T11:33:32.578 INFO:teuthology.orchestra.run.smithi072.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.7 MB/s | 217 kB 00:00 2024-03-29T11:33:32.622 INFO:teuthology.orchestra.run.smithi089.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 214 kB/s | 32 kB 00:00 2024-03-29T11:33:32.864 INFO:teuthology.orchestra.run.smithi089.stdout:(9/13): python3-kubernetes-26.1.0-2.el9.noarch. 4.2 MB/s | 1.0 MB 00:00 2024-03-29T11:33:32.923 INFO:teuthology.orchestra.run.smithi089.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-03-29T11:33:32.981 INFO:teuthology.orchestra.run.smithi089.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-03-29T11:33:33.046 INFO:teuthology.orchestra.run.smithi072.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 32 MB/s | 16 MB 00:00 2024-03-29T11:33:33.222 INFO:teuthology.orchestra.run.smithi072.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 803 kB 00:00 2024-03-29T11:33:33.347 INFO:teuthology.orchestra.run.smithi072.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 650 kB/s | 195 kB 00:00 2024-03-29T11:33:33.989 INFO:teuthology.orchestra.run.smithi072.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-1 3.6 MB/s | 7.4 MB 00:02 2024-03-29T11:33:34.197 INFO:teuthology.orchestra.run.smithi072.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 251 kB/s | 245 kB 00:00 2024-03-29T11:33:34.200 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:34.201 INFO:teuthology.orchestra.run.smithi072.stdout:Total 14 MB/s | 37 MB 00:02 2024-03-29T11:33:34.243 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-03-29T11:33:34.268 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-03-29T11:33:34.268 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-03-29T11:33:34.625 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-03-29T11:33:34.625 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-03-29T11:33:35.327 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-03-29T11:33:35.412 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:33:35.453 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:33:35.553 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:33:35.592 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:33:35.769 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:33:35.808 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:33:35.905 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:33:36.549 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:33:36.670 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:33:37.911 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:33:38.950 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:33:38.982 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:33:39.731 INFO:teuthology.orchestra.run.smithi089.stdout:(12/13): python3-certifi-2021.10.8-2.el9.noarch 2.1 kB/s | 15 kB 00:07 2024-03-29T11:33:39.847 INFO:teuthology.orchestra.run.smithi089.stdout:(13/13): python3-google-auth-2.28.2-1.el9.noarc 28 kB/s | 203 kB 00:07 2024-03-29T11:33:39.849 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:39.850 INFO:teuthology.orchestra.run.smithi089.stdout:Total 271 kB/s | 2.0 MB 00:07 2024-03-29T11:33:39.850 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-03-29T11:33:39.865 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-03-29T11:33:39.865 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-03-29T11:33:39.962 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-03-29T11:33:39.963 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-03-29T11:33:40.466 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-03-29T11:33:40.634 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:33:40.690 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:33:40.746 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:33:40.783 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:33:40.824 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:33:40.917 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:33:41.026 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:33:41.076 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:33:41.786 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:33:41.832 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:33:41.866 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:33:41.914 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:33:41.936 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:33:42.698 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:33:42.698 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:33:42.698 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:33:42.698 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:33:42.698 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:33:42.699 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:33:42.699 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:33:42.699 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:33:42.699 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:33:42.699 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:33:42.699 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:33:42.699 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:33:42.699 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:33:43.045 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:33:43.046 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:33:43.046 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:33:43.046 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:33:43.046 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:33:43.046 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:33:43.046 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:33:43.046 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:33:43.046 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:33:43.046 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:33:43.046 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:33:43.046 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:33:43.057 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:33:43.057 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:43.057 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-03-29T11:33:43.057 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:43.057 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:33:43.057 INFO:teuthology.orchestra.run.smithi089.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:33:43.057 INFO:teuthology.orchestra.run.smithi089.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:33:43.057 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:33:43.057 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:33:43.057 INFO:teuthology.orchestra.run.smithi089.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:33:43.057 INFO:teuthology.orchestra.run.smithi089.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:33:43.057 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:33:43.058 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:33:43.058 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:33:43.058 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:33:43.058 INFO:teuthology.orchestra.run.smithi089.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:33:43.058 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:43.058 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-29T11:33:43.330 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:33:43.485 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:33:43.485 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:33:43.485 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-03-29T11:33:43.485 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:43.485 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:33:43.485 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:33:43.485 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:33:43.485 INFO:teuthology.orchestra.run.smithi072.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:33:43.485 INFO:teuthology.orchestra.run.smithi072.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:33:43.486 INFO:teuthology.orchestra.run.smithi072.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:33:43.486 INFO:teuthology.orchestra.run.smithi072.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:33:43.486 INFO:teuthology.orchestra.run.smithi072.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:33:43.486 INFO:teuthology.orchestra.run.smithi072.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:33:43.486 INFO:teuthology.orchestra.run.smithi072.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:33:43.486 INFO:teuthology.orchestra.run.smithi072.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:33:43.486 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:33:43.486 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-29T11:33:43.894 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:02:24 ago on Fri 29 Mar 2024 11:31:19 AM UTC. 2024-03-29T11:33:43.912 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:33:44.047 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-29T11:33:44.047 INFO:teuthology.orchestra.run.smithi089.stdout:======================================================================================== 2024-03-29T11:33:44.047 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-03-29T11:33:44.047 INFO:teuthology.orchestra.run.smithi089.stdout:======================================================================================== 2024-03-29T11:33:44.047 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-03-29T11:33:44.047 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:33:44.047 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-03-29T11:33:44.047 INFO:teuthology.orchestra.run.smithi089.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:33:44.047 INFO:teuthology.orchestra.run.smithi089.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:33:44.047 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:33:44.048 INFO:teuthology.orchestra.run.smithi089.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:33:44.048 INFO:teuthology.orchestra.run.smithi089.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:33:44.048 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:44.048 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-03-29T11:33:44.048 INFO:teuthology.orchestra.run.smithi089.stdout:======================================================================================== 2024-03-29T11:33:44.048 INFO:teuthology.orchestra.run.smithi089.stdout:Install 6 Packages 2024-03-29T11:33:44.048 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:44.049 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 6.8 M 2024-03-29T11:33:44.049 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 33 M 2024-03-29T11:33:44.049 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-03-29T11:33:44.475 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:27 ago on Fri 29 Mar 2024 11:31:17 AM UTC. 2024-03-29T11:33:44.556 INFO:teuthology.orchestra.run.smithi089.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2024-03-29T11:33:44.611 INFO:teuthology.orchestra.run.smithi089.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 673 kB/s | 248 kB 00:00 2024-03-29T11:33:44.630 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-29T11:33:44.639 INFO:teuthology.orchestra.run.smithi072.stdout:======================================================================================== 2024-03-29T11:33:44.639 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2024-03-29T11:33:44.639 INFO:teuthology.orchestra.run.smithi072.stdout:======================================================================================== 2024-03-29T11:33:44.639 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-03-29T11:33:44.639 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:33:44.639 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2024-03-29T11:33:44.640 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:33:44.640 INFO:teuthology.orchestra.run.smithi072.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:33:44.640 INFO:teuthology.orchestra.run.smithi072.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:33:44.640 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:33:44.640 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:33:44.640 INFO:teuthology.orchestra.run.smithi072.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:33:44.640 INFO:teuthology.orchestra.run.smithi072.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:33:44.640 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:33:44.641 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:33:44.641 INFO:teuthology.orchestra.run.smithi072.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:33:44.641 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:33:44.641 INFO:teuthology.orchestra.run.smithi072.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:33:44.641 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:33:44.641 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-03-29T11:33:44.641 INFO:teuthology.orchestra.run.smithi072.stdout:======================================================================================== 2024-03-29T11:33:44.642 INFO:teuthology.orchestra.run.smithi072.stdout:Install 13 Packages 2024-03-29T11:33:44.642 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:33:44.644 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 2.0 M 2024-03-29T11:33:44.644 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 26 M 2024-03-29T11:33:44.644 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-03-29T11:33:44.719 INFO:teuthology.orchestra.run.smithi089.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 289 kB/s | 138 kB 00:00 2024-03-29T11:33:44.802 INFO:teuthology.orchestra.run.smithi089.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 300 kB/s | 58 kB 00:00 2024-03-29T11:33:44.828 INFO:teuthology.orchestra.run.smithi089.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 473 kB/s | 51 kB 00:00 2024-03-29T11:33:44.853 INFO:teuthology.orchestra.run.smithi089.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.8 MB/s | 548 kB 00:00 2024-03-29T11:33:44.856 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:44.857 INFO:teuthology.orchestra.run.smithi089.stdout:Total 8.4 MB/s | 6.8 MB 00:00 2024-03-29T11:33:44.870 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-03-29T11:33:44.880 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-03-29T11:33:44.880 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-03-29T11:33:44.970 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-03-29T11:33:44.970 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-03-29T11:33:44.971 INFO:teuthology.orchestra.run.smithi072.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 140 kB/s | 18 kB 00:00 2024-03-29T11:33:44.996 INFO:teuthology.orchestra.run.smithi072.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 158 kB/s | 24 kB 00:00 2024-03-29T11:33:45.072 INFO:teuthology.orchestra.run.smithi072.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.7 MB/s | 174 kB 00:00 2024-03-29T11:33:45.098 INFO:teuthology.orchestra.run.smithi072.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 132 kB 00:00 2024-03-29T11:33:45.125 INFO:teuthology.orchestra.run.smithi072.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.9 MB/s | 211 kB 00:00 2024-03-29T11:33:45.150 INFO:teuthology.orchestra.run.smithi072.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 823 kB/s | 43 kB 00:00 2024-03-29T11:33:45.176 INFO:teuthology.orchestra.run.smithi072.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 156 kB/s | 52 kB 00:00 2024-03-29T11:33:45.277 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-03-29T11:33:45.318 INFO:teuthology.orchestra.run.smithi072.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 167 kB/s | 32 kB 00:00 2024-03-29T11:33:45.352 INFO:teuthology.orchestra.run.smithi072.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 74 kB/s | 15 kB 00:00 2024-03-29T11:33:45.461 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:33:45.477 INFO:teuthology.orchestra.run.smithi072.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 795 kB/s | 203 kB 00:00 2024-03-29T11:33:45.505 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:33:45.578 INFO:teuthology.orchestra.run.smithi072.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 3.9 MB/s | 1.0 MB 00:00 2024-03-29T11:33:45.612 INFO:teuthology.orchestra.run.smithi072.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 228 kB/s | 59 kB 00:00 2024-03-29T11:33:45.637 INFO:teuthology.orchestra.run.smithi072.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 564 kB/s | 90 kB 00:00 2024-03-29T11:33:45.641 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:45.641 INFO:teuthology.orchestra.run.smithi072.stdout:Total 2.0 MB/s | 2.0 MB 00:01 2024-03-29T11:33:45.641 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-03-29T11:33:45.654 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-03-29T11:33:45.654 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-03-29T11:33:45.743 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-03-29T11:33:45.743 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-03-29T11:33:46.020 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:33:46.117 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:33:46.186 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:33:46.209 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:33:46.256 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-03-29T11:33:46.426 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:33:46.482 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:33:46.563 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:33:46.602 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:33:46.644 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:33:46.755 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:33:46.863 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:33:46.924 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:33:47.019 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:33:47.020 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:33:47.020 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:33:47.020 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:33:47.020 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:33:47.020 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:33:47.306 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:33:47.306 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:47.306 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-03-29T11:33:47.306 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:47.306 INFO:teuthology.orchestra.run.smithi089.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:33:47.306 INFO:teuthology.orchestra.run.smithi089.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:33:47.307 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:33:47.307 INFO:teuthology.orchestra.run.smithi089.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:33:47.307 INFO:teuthology.orchestra.run.smithi089.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:33:47.307 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:47.307 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-29T11:33:47.560 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-fuse 2024-03-29T11:33:47.637 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:33:47.694 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:33:47.728 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:33:47.776 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:33:47.808 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:33:48.118 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:02:29 ago on Fri 29 Mar 2024 11:31:19 AM UTC. 2024-03-29T11:33:48.267 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-29T11:33:48.267 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-29T11:33:48.267 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-03-29T11:33:48.267 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-29T11:33:48.267 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-03-29T11:33:48.267 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:33:48.268 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-03-29T11:33:48.268 INFO:teuthology.orchestra.run.smithi089.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:33:48.268 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:48.268 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-03-29T11:33:48.268 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-29T11:33:48.268 INFO:teuthology.orchestra.run.smithi089.stdout:Install 2 Packages 2024-03-29T11:33:48.268 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:48.269 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 964 k 2024-03-29T11:33:48.269 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 2.8 M 2024-03-29T11:33:48.269 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-03-29T11:33:48.408 INFO:teuthology.orchestra.run.smithi089.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 566 kB/s | 79 kB 00:00 2024-03-29T11:33:48.650 INFO:teuthology.orchestra.run.smithi089.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 2.3 MB/s | 885 kB 00:00 2024-03-29T11:33:48.651 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:48.651 INFO:teuthology.orchestra.run.smithi089.stdout:Total 2.5 MB/s | 964 kB 00:00 2024-03-29T11:33:48.651 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-03-29T11:33:48.671 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-03-29T11:33:48.672 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-03-29T11:33:48.747 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-03-29T11:33:48.747 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-03-29T11:33:48.855 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:33:48.855 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:33:48.855 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:33:48.855 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:33:48.855 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:33:48.855 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:33:48.856 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:33:48.856 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:33:48.856 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:33:48.856 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:33:48.856 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:33:48.856 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:33:48.856 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:33:48.928 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-03-29T11:33:49.067 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:33:49.143 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:33:49.204 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:33:49.204 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:33:49.204 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-03-29T11:33:49.204 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:49.204 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:33:49.204 INFO:teuthology.orchestra.run.smithi072.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:33:49.204 INFO:teuthology.orchestra.run.smithi072.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:33:49.204 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:33:49.204 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:33:49.205 INFO:teuthology.orchestra.run.smithi072.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:33:49.205 INFO:teuthology.orchestra.run.smithi072.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:33:49.205 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:33:49.205 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:33:49.205 INFO:teuthology.orchestra.run.smithi072.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:33:49.205 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:33:49.205 INFO:teuthology.orchestra.run.smithi072.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:33:49.205 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:33:49.205 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-29T11:33:49.529 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:33:49.842 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:33:49.842 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:33:50.098 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:33 ago on Fri 29 Mar 2024 11:31:17 AM UTC. 2024-03-29T11:33:50.173 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:33:50.173 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:50.173 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-03-29T11:33:50.174 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:33:50.174 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:50.174 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-29T11:33:50.254 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-29T11:33:50.255 INFO:teuthology.orchestra.run.smithi072.stdout:======================================================================================== 2024-03-29T11:33:50.255 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2024-03-29T11:33:50.255 INFO:teuthology.orchestra.run.smithi072.stdout:======================================================================================== 2024-03-29T11:33:50.255 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-03-29T11:33:50.255 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:33:50.255 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2024-03-29T11:33:50.255 INFO:teuthology.orchestra.run.smithi072.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:33:50.255 INFO:teuthology.orchestra.run.smithi072.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:33:50.256 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:33:50.256 INFO:teuthology.orchestra.run.smithi072.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:33:50.256 INFO:teuthology.orchestra.run.smithi072.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:33:50.256 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:33:50.256 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-03-29T11:33:50.256 INFO:teuthology.orchestra.run.smithi072.stdout:======================================================================================== 2024-03-29T11:33:50.256 INFO:teuthology.orchestra.run.smithi072.stdout:Install 6 Packages 2024-03-29T11:33:50.256 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:33:50.257 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 6.8 M 2024-03-29T11:33:50.257 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 33 M 2024-03-29T11:33:50.257 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-03-29T11:33:50.500 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-volume 2024-03-29T11:33:50.871 INFO:teuthology.orchestra.run.smithi072.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-03-29T11:33:51.081 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:02:32 ago on Fri 29 Mar 2024 11:31:19 AM UTC. 2024-03-29T11:33:51.113 INFO:teuthology.orchestra.run.smithi072.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 323 kB/s | 138 kB 00:00 2024-03-29T11:33:51.215 INFO:teuthology.orchestra.run.smithi072.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.6 MB/s | 548 kB 00:00 2024-03-29T11:33:51.243 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-29T11:33:51.243 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-29T11:33:51.244 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-03-29T11:33:51.244 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-29T11:33:51.244 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-03-29T11:33:51.244 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:33:51.244 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-03-29T11:33:51.244 INFO:teuthology.orchestra.run.smithi089.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:33:51.244 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:51.244 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-03-29T11:33:51.244 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-29T11:33:51.244 INFO:teuthology.orchestra.run.smithi089.stdout:Install 2 Packages 2024-03-29T11:33:51.244 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:51.245 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 332 k 2024-03-29T11:33:51.245 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 1.5 M 2024-03-29T11:33:51.245 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-03-29T11:33:51.332 INFO:teuthology.orchestra.run.smithi072.stdout:(4/6): python3-babel-2.9.1-2.el9.noarch.rpm 8.9 MB/s | 5.8 MB 00:00 2024-03-29T11:33:51.358 INFO:teuthology.orchestra.run.smithi072.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 237 kB/s | 58 kB 00:00 2024-03-29T11:33:51.365 INFO:teuthology.orchestra.run.smithi089.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 576 kB/s | 69 kB 00:00 2024-03-29T11:33:51.391 INFO:teuthology.orchestra.run.smithi072.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 290 kB/s | 51 kB 00:00 2024-03-29T11:33:51.394 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:51.394 INFO:teuthology.orchestra.run.smithi072.stdout:Total 6.0 MB/s | 6.8 MB 00:01 2024-03-29T11:33:51.407 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-03-29T11:33:51.417 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-03-29T11:33:51.418 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-03-29T11:33:51.505 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-03-29T11:33:51.505 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-03-29T11:33:51.506 INFO:teuthology.orchestra.run.smithi089.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 1.0 MB/s | 264 kB 00:00 2024-03-29T11:33:51.508 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:51.509 INFO:teuthology.orchestra.run.smithi089.stdout:Total 1.2 MB/s | 332 kB 00:00 2024-03-29T11:33:51.509 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-03-29T11:33:51.518 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-03-29T11:33:51.518 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-03-29T11:33:51.556 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-03-29T11:33:51.556 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-03-29T11:33:51.706 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-03-29T11:33:51.803 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-03-29T11:33:51.893 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:33:51.926 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:33:51.944 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:33:51.944 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:51.944 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:33:51.944 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:51.976 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:33:52.028 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:33:52.547 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:33:52.643 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:33:52.705 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:33:52.737 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:33:52.739 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:33:53.072 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:33:53.072 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:53.072 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-03-29T11:33:53.072 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:53.073 INFO:teuthology.orchestra.run.smithi089.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:33:53.073 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:53.073 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-29T11:33:53.432 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install librados-devel 2024-03-29T11:33:53.836 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:33:53.837 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:33:53.837 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:33:53.837 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:33:53.837 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:33:53.837 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:33:54.015 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:02:35 ago on Fri 29 Mar 2024 11:31:19 AM UTC. 2024-03-29T11:33:54.129 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:33:54.129 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:33:54.129 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-03-29T11:33:54.129 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:54.129 INFO:teuthology.orchestra.run.smithi072.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:33:54.129 INFO:teuthology.orchestra.run.smithi072.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:33:54.129 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:33:54.129 INFO:teuthology.orchestra.run.smithi072.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:33:54.130 INFO:teuthology.orchestra.run.smithi072.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:33:54.130 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:33:54.130 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-29T11:33:54.164 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-29T11:33:54.165 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-29T11:33:54.165 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repo Size 2024-03-29T11:33:54.165 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-29T11:33:54.165 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-03-29T11:33:54.165 INFO:teuthology.orchestra.run.smithi089.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:33:54.165 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:54.165 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-03-29T11:33:54.165 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-29T11:33:54.165 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-03-29T11:33:54.165 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:54.166 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 127 k 2024-03-29T11:33:54.166 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 456 k 2024-03-29T11:33:54.166 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-03-29T11:33:54.475 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-fuse 2024-03-29T11:33:54.514 INFO:teuthology.orchestra.run.smithi089.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 364 kB/s | 127 kB 00:00 2024-03-29T11:33:54.515 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:54.515 INFO:teuthology.orchestra.run.smithi089.stdout:Total 363 kB/s | 127 kB 00:00 2024-03-29T11:33:54.515 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-03-29T11:33:54.520 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-03-29T11:33:54.520 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-03-29T11:33:54.577 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-03-29T11:33:54.577 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-03-29T11:33:54.732 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-03-29T11:33:54.837 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:55.048 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:38 ago on Fri 29 Mar 2024 11:31:17 AM UTC. 2024-03-29T11:33:55.200 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-29T11:33:55.201 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-29T11:33:55.201 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2024-03-29T11:33:55.201 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-29T11:33:55.201 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-03-29T11:33:55.201 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:33:55.201 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2024-03-29T11:33:55.201 INFO:teuthology.orchestra.run.smithi072.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:33:55.202 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:33:55.202 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-03-29T11:33:55.202 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-29T11:33:55.202 INFO:teuthology.orchestra.run.smithi072.stdout:Install 2 Packages 2024-03-29T11:33:55.202 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:33:55.202 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 964 k 2024-03-29T11:33:55.202 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 2.8 M 2024-03-29T11:33:55.202 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-03-29T11:33:55.275 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:55.332 INFO:teuthology.orchestra.run.smithi072.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 606 kB/s | 79 kB 00:00 2024-03-29T11:33:55.532 INFO:teuthology.orchestra.run.smithi072.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 2.6 MB/s | 885 kB 00:00 2024-03-29T11:33:55.534 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:55.534 INFO:teuthology.orchestra.run.smithi072.stdout:Total 2.8 MB/s | 964 kB 00:00 2024-03-29T11:33:55.534 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-03-29T11:33:55.555 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-03-29T11:33:55.555 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-03-29T11:33:55.571 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:55.571 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:55.571 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-03-29T11:33:55.571 INFO:teuthology.orchestra.run.smithi089.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:55.572 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:55.572 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-29T11:33:55.627 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-03-29T11:33:55.628 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-03-29T11:33:55.806 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-03-29T11:33:55.815 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install libcephfs2 2024-03-29T11:33:55.955 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:33:56.035 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:33:56.390 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:02:37 ago on Fri 29 Mar 2024 11:31:19 AM UTC. 2024-03-29T11:33:56.491 INFO:teuthology.orchestra.run.smithi089.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:56.547 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-29T11:33:56.548 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-03-29T11:33:56.549 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-29T11:33:56.630 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install libcephfs-devel 2024-03-29T11:33:56.754 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:33:56.755 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:33:57.065 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:33:57.065 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:33:57.065 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-03-29T11:33:57.065 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:33:57.065 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:33:57.065 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-29T11:33:57.197 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:02:38 ago on Fri 29 Mar 2024 11:31:19 AM UTC. 2024-03-29T11:33:57.349 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-29T11:33:57.349 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-29T11:33:57.349 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repo Size 2024-03-29T11:33:57.350 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-29T11:33:57.350 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-03-29T11:33:57.350 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:33:57.350 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:57.350 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-03-29T11:33:57.350 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-29T11:33:57.350 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-03-29T11:33:57.350 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:57.350 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 31 k 2024-03-29T11:33:57.350 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 140 k 2024-03-29T11:33:57.350 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-03-29T11:33:57.420 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-volume 2024-03-29T11:33:57.543 INFO:teuthology.orchestra.run.smithi089.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 162 kB/s | 31 kB 00:00 2024-03-29T11:33:57.545 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:57.545 INFO:teuthology.orchestra.run.smithi089.stdout:Total 160 kB/s | 31 kB 00:00 2024-03-29T11:33:57.546 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-03-29T11:33:57.547 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-03-29T11:33:57.548 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-03-29T11:33:57.572 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-03-29T11:33:57.572 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-03-29T11:33:57.700 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-03-29T11:33:57.791 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:57.992 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:40 ago on Fri 29 Mar 2024 11:31:17 AM UTC. 2024-03-29T11:33:58.150 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-29T11:33:58.151 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-29T11:33:58.151 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2024-03-29T11:33:58.151 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-29T11:33:58.151 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-03-29T11:33:58.151 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:33:58.151 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2024-03-29T11:33:58.151 INFO:teuthology.orchestra.run.smithi072.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:33:58.151 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:33:58.151 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-03-29T11:33:58.151 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-29T11:33:58.151 INFO:teuthology.orchestra.run.smithi072.stdout:Install 2 Packages 2024-03-29T11:33:58.151 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:33:58.152 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 332 k 2024-03-29T11:33:58.152 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 1.5 M 2024-03-29T11:33:58.152 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-03-29T11:33:58.164 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:58.263 INFO:teuthology.orchestra.run.smithi072.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 626 kB/s | 69 kB 00:00 2024-03-29T11:33:58.421 INFO:teuthology.orchestra.run.smithi072.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 982 kB/s | 264 kB 00:00 2024-03-29T11:33:58.422 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:58.422 INFO:teuthology.orchestra.run.smithi072.stdout:Total 1.2 MB/s | 332 kB 00:00 2024-03-29T11:33:58.423 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-03-29T11:33:58.428 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:58.428 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:58.429 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-03-29T11:33:58.429 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:58.429 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:33:58.429 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-29T11:33:58.431 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-03-29T11:33:58.431 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-03-29T11:33:58.473 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-03-29T11:33:58.473 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-03-29T11:33:58.587 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install librados2 2024-03-29T11:33:58.625 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-03-29T11:33:58.807 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:33:58.853 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:33:58.868 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:33:58.868 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:58.868 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:33:58.868 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:33:59.147 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:02:40 ago on Fri 29 Mar 2024 11:31:19 AM UTC. 2024-03-29T11:33:59.243 INFO:teuthology.orchestra.run.smithi089.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:59.295 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-29T11:33:59.296 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-03-29T11:33:59.296 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-29T11:33:59.340 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install librbd1 2024-03-29T11:33:59.683 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:33:59.913 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:02:40 ago on Fri 29 Mar 2024 11:31:19 AM UTC. 2024-03-29T11:34:00.000 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:34:00.000 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:00.000 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-03-29T11:34:00.000 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:34:00.000 INFO:teuthology.orchestra.run.smithi072.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:34:00.000 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:00.000 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-29T11:34:00.009 INFO:teuthology.orchestra.run.smithi089.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:34:00.061 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-29T11:34:00.062 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-03-29T11:34:00.062 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-29T11:34:00.117 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install python3-rados 2024-03-29T11:34:00.409 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install librados-devel 2024-03-29T11:34:00.682 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:02:41 ago on Fri 29 Mar 2024 11:31:19 AM UTC. 2024-03-29T11:34:00.783 INFO:teuthology.orchestra.run.smithi089.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:34:00.835 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-29T11:34:00.836 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-03-29T11:34:00.836 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-29T11:34:00.884 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install python3-rgw 2024-03-29T11:34:00.987 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:43 ago on Fri 29 Mar 2024 11:31:17 AM UTC. 2024-03-29T11:34:01.139 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-29T11:34:01.139 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-29T11:34:01.139 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repo Size 2024-03-29T11:34:01.140 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-29T11:34:01.140 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-03-29T11:34:01.140 INFO:teuthology.orchestra.run.smithi072.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:34:01.140 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:01.140 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-03-29T11:34:01.140 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-29T11:34:01.140 INFO:teuthology.orchestra.run.smithi072.stdout:Install 1 Package 2024-03-29T11:34:01.140 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:01.141 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 127 k 2024-03-29T11:34:01.141 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 456 k 2024-03-29T11:34:01.141 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-03-29T11:34:01.446 INFO:teuthology.orchestra.run.smithi072.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 415 kB/s | 127 kB 00:00 2024-03-29T11:34:01.447 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:34:01.447 INFO:teuthology.orchestra.run.smithi072.stdout:Total 413 kB/s | 127 kB 00:00 2024-03-29T11:34:01.447 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-03-29T11:34:01.449 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:02:42 ago on Fri 29 Mar 2024 11:31:19 AM UTC. 2024-03-29T11:34:01.452 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-03-29T11:34:01.452 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-03-29T11:34:01.508 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-03-29T11:34:01.508 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-03-29T11:34:01.548 INFO:teuthology.orchestra.run.smithi089.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:34:01.600 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-29T11:34:01.601 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-03-29T11:34:01.601 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-29T11:34:01.654 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-03-29T11:34:01.663 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install python3-cephfs 2024-03-29T11:34:01.763 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:02.234 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:02:43 ago on Fri 29 Mar 2024 11:31:19 AM UTC. 2024-03-29T11:34:02.280 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:02.337 INFO:teuthology.orchestra.run.smithi089.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:34:02.390 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-29T11:34:02.391 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-03-29T11:34:02.392 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-29T11:34:02.450 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install python3-rbd 2024-03-29T11:34:02.574 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:02.574 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:02.574 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-03-29T11:34:02.574 INFO:teuthology.orchestra.run.smithi072.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:02.574 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:02.574 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-29T11:34:02.883 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install libcephfs2 2024-03-29T11:34:03.017 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:02:44 ago on Fri 29 Mar 2024 11:31:19 AM UTC. 2024-03-29T11:34:03.114 INFO:teuthology.orchestra.run.smithi089.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:34:03.167 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-29T11:34:03.168 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-03-29T11:34:03.168 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-29T11:34:03.222 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install rbd-fuse 2024-03-29T11:34:03.453 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:46 ago on Fri 29 Mar 2024 11:31:17 AM UTC. 2024-03-29T11:34:03.552 INFO:teuthology.orchestra.run.smithi072.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:34:03.609 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-29T11:34:03.610 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2024-03-29T11:34:03.610 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-29T11:34:03.691 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install libcephfs-devel 2024-03-29T11:34:03.776 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:02:44 ago on Fri 29 Mar 2024 11:31:19 AM UTC. 2024-03-29T11:34:03.923 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-29T11:34:03.924 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-29T11:34:03.924 INFO:teuthology.orchestra.run.smithi089.stdout: Package Architecture Version Repository Size 2024-03-29T11:34:03.924 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-29T11:34:03.924 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-03-29T11:34:03.924 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:34:03.924 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:34:03.925 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-03-29T11:34:03.925 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-29T11:34:03.925 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-03-29T11:34:03.925 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:34:03.925 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 85 k 2024-03-29T11:34:03.925 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 231 k 2024-03-29T11:34:03.925 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-03-29T11:34:04.205 INFO:teuthology.orchestra.run.smithi089.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 305 kB/s | 85 kB 00:00 2024-03-29T11:34:04.205 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:34:04.206 INFO:teuthology.orchestra.run.smithi089.stdout:Total 303 kB/s | 85 kB 00:00 2024-03-29T11:34:04.206 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-03-29T11:34:04.213 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-03-29T11:34:04.213 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-03-29T11:34:04.258 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:47 ago on Fri 29 Mar 2024 11:31:17 AM UTC. 2024-03-29T11:34:04.267 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-03-29T11:34:04.268 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-03-29T11:34:04.404 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-03-29T11:34:04.409 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-29T11:34:04.410 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-29T11:34:04.410 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repo Size 2024-03-29T11:34:04.410 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-29T11:34:04.410 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-03-29T11:34:04.410 INFO:teuthology.orchestra.run.smithi072.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:34:04.410 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:04.410 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-03-29T11:34:04.410 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-29T11:34:04.410 INFO:teuthology.orchestra.run.smithi072.stdout:Install 1 Package 2024-03-29T11:34:04.411 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:04.411 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 31 k 2024-03-29T11:34:04.411 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 140 k 2024-03-29T11:34:04.411 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-03-29T11:34:04.503 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:04.647 INFO:teuthology.orchestra.run.smithi072.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 132 kB/s | 31 kB 00:00 2024-03-29T11:34:04.647 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:34:04.647 INFO:teuthology.orchestra.run.smithi072.stdout:Total 132 kB/s | 31 kB 00:00 2024-03-29T11:34:04.648 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-03-29T11:34:04.651 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-03-29T11:34:04.651 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-03-29T11:34:04.682 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-03-29T11:34:04.683 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-03-29T11:34:04.814 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-03-29T11:34:04.887 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:04.914 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:05.179 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:05.180 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:34:05.180 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-03-29T11:34:05.180 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:05.180 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:34:05.180 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-29T11:34:05.325 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:05.381 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install rbd-mirror 2024-03-29T11:34:05.574 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:05.574 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:05.574 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-03-29T11:34:05.574 INFO:teuthology.orchestra.run.smithi072.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:05.574 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:05.574 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-29T11:34:05.749 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install librados2 2024-03-29T11:34:05.952 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:02:46 ago on Fri 29 Mar 2024 11:31:19 AM UTC. 2024-03-29T11:34:06.108 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-29T11:34:06.109 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-29T11:34:06.109 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repo Size 2024-03-29T11:34:06.109 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-29T11:34:06.109 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-03-29T11:34:06.109 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:34:06.109 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:34:06.109 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-03-29T11:34:06.109 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-29T11:34:06.109 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-03-29T11:34:06.109 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:34:06.109 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 3.1 M 2024-03-29T11:34:06.109 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 13 M 2024-03-29T11:34:06.109 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-03-29T11:34:06.309 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:49 ago on Fri 29 Mar 2024 11:31:17 AM UTC. 2024-03-29T11:34:06.406 INFO:teuthology.orchestra.run.smithi072.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:34:06.458 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-29T11:34:06.459 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2024-03-29T11:34:06.459 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-29T11:34:06.504 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install librbd1 2024-03-29T11:34:06.635 INFO:teuthology.orchestra.run.smithi089.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 5.9 MB/s | 3.1 MB 00:00 2024-03-29T11:34:06.636 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:34:06.636 INFO:teuthology.orchestra.run.smithi089.stdout:Total 5.9 MB/s | 3.1 MB 00:00 2024-03-29T11:34:06.636 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-03-29T11:34:06.645 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-03-29T11:34:06.645 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-03-29T11:34:06.719 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-03-29T11:34:06.719 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-03-29T11:34:07.074 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:50 ago on Fri 29 Mar 2024 11:31:17 AM UTC. 2024-03-29T11:34:07.156 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-03-29T11:34:07.171 INFO:teuthology.orchestra.run.smithi072.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:34:07.224 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-29T11:34:07.225 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2024-03-29T11:34:07.225 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-29T11:34:07.227 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:07.251 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:07.251 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:34:07.251 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:34:07.251 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:34:07.251 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:34:07.251 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:34:07.277 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install python3-rados 2024-03-29T11:34:07.842 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:50 ago on Fri 29 Mar 2024 11:31:17 AM UTC. 2024-03-29T11:34:07.940 INFO:teuthology.orchestra.run.smithi072.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:34:07.992 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-29T11:34:07.992 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2024-03-29T11:34:07.993 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-29T11:34:08.048 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install python3-rgw 2024-03-29T11:34:08.355 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:08.355 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:34:08.355 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-03-29T11:34:08.355 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:08.355 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:34:08.355 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-29T11:34:08.579 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install rbd-nbd 2024-03-29T11:34:08.615 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:51 ago on Fri 29 Mar 2024 11:31:17 AM UTC. 2024-03-29T11:34:08.714 INFO:teuthology.orchestra.run.smithi072.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:34:08.766 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-29T11:34:08.767 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2024-03-29T11:34:08.767 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-29T11:34:08.819 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install python3-cephfs 2024-03-29T11:34:09.147 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:02:50 ago on Fri 29 Mar 2024 11:31:19 AM UTC. 2024-03-29T11:34:09.298 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-03-29T11:34:09.299 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-29T11:34:09.299 INFO:teuthology.orchestra.run.smithi089.stdout: Package Architecture Version Repository Size 2024-03-29T11:34:09.299 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-29T11:34:09.299 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-03-29T11:34:09.299 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:34:09.299 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:34:09.299 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-03-29T11:34:09.299 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-03-29T11:34:09.299 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-03-29T11:34:09.299 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:34:09.299 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 171 k 2024-03-29T11:34:09.300 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 494 k 2024-03-29T11:34:09.300 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-03-29T11:34:09.388 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:52 ago on Fri 29 Mar 2024 11:31:17 AM UTC. 2024-03-29T11:34:09.485 INFO:teuthology.orchestra.run.smithi072.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:34:09.536 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-29T11:34:09.537 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2024-03-29T11:34:09.537 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-29T11:34:09.587 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install python3-rbd 2024-03-29T11:34:09.647 INFO:teuthology.orchestra.run.smithi089.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 494 kB/s | 171 kB 00:00 2024-03-29T11:34:09.648 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:34:09.648 INFO:teuthology.orchestra.run.smithi089.stdout:Total 492 kB/s | 171 kB 00:00 2024-03-29T11:34:09.648 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-03-29T11:34:09.656 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-03-29T11:34:09.656 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-03-29T11:34:09.711 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-03-29T11:34:09.711 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-03-29T11:34:09.860 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-03-29T11:34:09.979 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:10.148 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:53 ago on Fri 29 Mar 2024 11:31:17 AM UTC. 2024-03-29T11:34:10.246 INFO:teuthology.orchestra.run.smithi072.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:34:10.297 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-29T11:34:10.298 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2024-03-29T11:34:10.298 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-29T11:34:10.344 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install rbd-fuse 2024-03-29T11:34:10.419 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:10.729 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:10.730 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:34:10.730 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-03-29T11:34:10.730 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:10.730 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:34:10.730 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-03-29T11:34:10.906 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:53 ago on Fri 29 Mar 2024 11:31:17 AM UTC. 2024-03-29T11:34:10.985 DEBUG:teuthology.parallel:result is None 2024-03-29T11:34:11.053 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-29T11:34:11.054 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-29T11:34:11.054 INFO:teuthology.orchestra.run.smithi072.stdout: Package Architecture Version Repository Size 2024-03-29T11:34:11.054 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-29T11:34:11.054 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-03-29T11:34:11.054 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:34:11.054 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:11.054 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-03-29T11:34:11.054 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-29T11:34:11.054 INFO:teuthology.orchestra.run.smithi072.stdout:Install 1 Package 2024-03-29T11:34:11.054 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:11.054 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 85 k 2024-03-29T11:34:11.054 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 231 k 2024-03-29T11:34:11.055 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-03-29T11:34:11.364 INFO:teuthology.orchestra.run.smithi072.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 276 kB/s | 85 kB 00:00 2024-03-29T11:34:11.367 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:34:11.368 INFO:teuthology.orchestra.run.smithi072.stdout:Total 275 kB/s | 85 kB 00:00 2024-03-29T11:34:11.368 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-03-29T11:34:11.372 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-03-29T11:34:11.374 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-03-29T11:34:11.429 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-03-29T11:34:11.429 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-03-29T11:34:11.566 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-03-29T11:34:11.675 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:12.117 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:12.414 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:12.414 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:12.415 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-03-29T11:34:12.415 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:12.415 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:12.415 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-29T11:34:12.720 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install rbd-mirror 2024-03-29T11:34:13.292 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:56 ago on Fri 29 Mar 2024 11:31:17 AM UTC. 2024-03-29T11:34:13.444 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-29T11:34:13.444 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-29T11:34:13.445 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repo Size 2024-03-29T11:34:13.445 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-29T11:34:13.445 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-03-29T11:34:13.445 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:34:13.445 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:13.445 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-03-29T11:34:13.445 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-29T11:34:13.445 INFO:teuthology.orchestra.run.smithi072.stdout:Install 1 Package 2024-03-29T11:34:13.445 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:13.445 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 3.1 M 2024-03-29T11:34:13.445 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 13 M 2024-03-29T11:34:13.445 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-03-29T11:34:14.324 INFO:teuthology.orchestra.run.smithi072.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 3.5 MB/s | 3.1 MB 00:00 2024-03-29T11:34:14.325 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:34:14.326 INFO:teuthology.orchestra.run.smithi072.stdout:Total 3.5 MB/s | 3.1 MB 00:00 2024-03-29T11:34:14.326 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-03-29T11:34:14.336 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-03-29T11:34:14.336 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-03-29T11:34:14.412 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-03-29T11:34:14.412 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-03-29T11:34:14.844 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-03-29T11:34:14.909 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:14.933 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:14.934 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:34:14.934 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:34:14.934 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:34:14.934 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:34:14.934 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:16.171 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:16.171 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:16.171 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-03-29T11:34:16.171 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:16.171 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:16.171 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-29T11:34:16.448 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install rbd-nbd 2024-03-29T11:34:17.023 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:03:00 ago on Fri 29 Mar 2024 11:31:17 AM UTC. 2024-03-29T11:34:17.171 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-29T11:34:17.172 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-29T11:34:17.172 INFO:teuthology.orchestra.run.smithi072.stdout: Package Architecture Version Repository Size 2024-03-29T11:34:17.172 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-29T11:34:17.172 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-03-29T11:34:17.172 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:34:17.173 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:17.173 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-03-29T11:34:17.173 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-29T11:34:17.173 INFO:teuthology.orchestra.run.smithi072.stdout:Install 1 Package 2024-03-29T11:34:17.173 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:17.173 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 171 k 2024-03-29T11:34:17.173 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 494 k 2024-03-29T11:34:17.173 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-03-29T11:34:17.419 INFO:teuthology.orchestra.run.smithi072.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 698 kB/s | 171 kB 00:00 2024-03-29T11:34:17.419 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:34:17.420 INFO:teuthology.orchestra.run.smithi072.stdout:Total 694 kB/s | 171 kB 00:00 2024-03-29T11:34:17.420 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-03-29T11:34:17.427 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-03-29T11:34:17.427 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-03-29T11:34:17.484 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-03-29T11:34:17.484 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-03-29T11:34:17.628 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-03-29T11:34:17.743 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:18.212 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:18.503 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:18.503 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:18.503 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-03-29T11:34:18.503 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:18.504 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:18.504 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-29T11:34:18.761 DEBUG:teuthology.parallel:result is None 2024-03-29T11:34:18.762 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:34:18.902 DEBUG:teuthology.orchestra.run.smithi072:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:34:18.923 INFO:teuthology.orchestra.run.smithi072.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:34:18.923 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:34:18.924 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:34:18.926 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:34:19.148 DEBUG:teuthology.orchestra.run.smithi089:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:34:19.173 INFO:teuthology.orchestra.run.smithi089.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:34:19.173 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:34:19.173 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:34:19.175 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T11:34:19.176 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-29T11:34:19.176 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:34:19.203 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-29T11:34:19.203 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:34:19.246 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T11:34:19.246 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-29T11:34:19.246 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:34:19.273 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:34:19.339 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-29T11:34:19.339 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:34:19.369 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:34:19.439 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T11:34:19.439 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-29T11:34:19.439 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:34:19.465 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:34:19.532 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-29T11:34:19.532 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:34:19.564 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:34:19.633 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T11:34:19.633 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-29T11:34:19.634 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:34:19.660 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:34:19.730 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-29T11:34:19.730 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:34:19.757 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:34:19.824 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T11:34:19.914 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T11:34:19.914 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 777 /var/log/ceph 2024-03-29T11:34:19.916 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 777 /var/log/ceph 2024-03-29T11:34:19.944 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T11:34:19.944 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:34:19.983 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:34:20.011 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T11:34:20.011 DEBUG:teuthology.orchestra.run.smithi072:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:34:20.050 DEBUG:teuthology.orchestra.run.smithi089:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:34:20.089 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T11:34:20.090 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T11:34:20.090 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622722', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622722', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi072.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBnkRsNyU0uNAD+2aatvNpmr+GleVEkLFEE6m7MMj06nj5J8vftU4nbNo8dSce5aVYqvPCwgkVTxgpesC/Q1+bI=', 'smithi089.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPxSfkA8KNtwlNRpzeShqCxAH0NIYQ6PieBELRqnTCPw6tYsU9YdIh3hFkW56jYhLxj9fCJLD06xQQkl7q+9VVY='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T11:34:20.090 DEBUG:teuthology.orchestra.run.smithi072:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:34:20.117 DEBUG:teuthology.orchestra.run.smithi089:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:34:20.134 DEBUG:teuthology.orchestra.run.smithi072:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:34:20.171 DEBUG:teuthology.orchestra.run.smithi089:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:34:20.202 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-29T11:34:20.202 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:34:20.251 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:34:20.251 DEBUG:teuthology.orchestra.run.smithi072:> stat /dev/vg_nvme/lv_1 2024-03-29T11:34:20.305 INFO:teuthology.orchestra.run.smithi072.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:34:20.305 INFO:teuthology.orchestra.run.smithi072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:34:20.305 INFO:teuthology.orchestra.run.smithi072.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-03-29T11:34:20.305 INFO:teuthology.orchestra.run.smithi072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:34:20.305 INFO:teuthology.orchestra.run.smithi072.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:34:20.305 INFO:teuthology.orchestra.run.smithi072.stdout:Access: 2024-03-29 11:34:15.183883800 +0000 2024-03-29T11:34:20.306 INFO:teuthology.orchestra.run.smithi072.stdout:Modify: 2024-03-29 11:30:17.909188091 +0000 2024-03-29T11:34:20.306 INFO:teuthology.orchestra.run.smithi072.stdout:Change: 2024-03-29 11:30:17.909188091 +0000 2024-03-29T11:34:20.306 INFO:teuthology.orchestra.run.smithi072.stdout: Birth: 2024-03-29 11:30:17.909188091 +0000 2024-03-29T11:34:20.306 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:34:20.370 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records in 2024-03-29T11:34:20.371 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records out 2024-03-29T11:34:20.371 INFO:teuthology.orchestra.run.smithi072.stderr:512 bytes copied, 0.000239252 s, 2.1 MB/s 2024-03-29T11:34:20.372 DEBUG:teuthology.orchestra.run.smithi072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:34:20.427 DEBUG:teuthology.orchestra.run.smithi072:> stat /dev/vg_nvme/lv_2 2024-03-29T11:34:20.483 INFO:teuthology.orchestra.run.smithi072.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:34:20.483 INFO:teuthology.orchestra.run.smithi072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:34:20.483 INFO:teuthology.orchestra.run.smithi072.stdout:Device: 5h/5d Inode: 682 Links: 1 2024-03-29T11:34:20.483 INFO:teuthology.orchestra.run.smithi072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:34:20.483 INFO:teuthology.orchestra.run.smithi072.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:34:20.483 INFO:teuthology.orchestra.run.smithi072.stdout:Access: 2024-03-29 11:34:15.183883800 +0000 2024-03-29T11:34:20.483 INFO:teuthology.orchestra.run.smithi072.stdout:Modify: 2024-03-29 11:30:18.273180217 +0000 2024-03-29T11:34:20.484 INFO:teuthology.orchestra.run.smithi072.stdout:Change: 2024-03-29 11:30:18.273180217 +0000 2024-03-29T11:34:20.484 INFO:teuthology.orchestra.run.smithi072.stdout: Birth: 2024-03-29 11:30:18.273180217 +0000 2024-03-29T11:34:20.484 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:34:20.548 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records in 2024-03-29T11:34:20.548 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records out 2024-03-29T11:34:20.548 INFO:teuthology.orchestra.run.smithi072.stderr:512 bytes copied, 0.000190265 s, 2.7 MB/s 2024-03-29T11:34:20.549 DEBUG:teuthology.orchestra.run.smithi072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:34:20.604 DEBUG:teuthology.orchestra.run.smithi072:> stat /dev/vg_nvme/lv_3 2024-03-29T11:34:20.660 INFO:teuthology.orchestra.run.smithi072.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:34:20.660 INFO:teuthology.orchestra.run.smithi072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:34:20.660 INFO:teuthology.orchestra.run.smithi072.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-29T11:34:20.660 INFO:teuthology.orchestra.run.smithi072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:34:20.660 INFO:teuthology.orchestra.run.smithi072.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:34:20.660 INFO:teuthology.orchestra.run.smithi072.stdout:Access: 2024-03-29 11:34:15.184883778 +0000 2024-03-29T11:34:20.661 INFO:teuthology.orchestra.run.smithi072.stdout:Modify: 2024-03-29 11:30:18.622172667 +0000 2024-03-29T11:34:20.661 INFO:teuthology.orchestra.run.smithi072.stdout:Change: 2024-03-29 11:30:18.622172667 +0000 2024-03-29T11:34:20.661 INFO:teuthology.orchestra.run.smithi072.stdout: Birth: 2024-03-29 11:30:18.622172667 +0000 2024-03-29T11:34:20.661 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:34:20.725 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records in 2024-03-29T11:34:20.725 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records out 2024-03-29T11:34:20.725 INFO:teuthology.orchestra.run.smithi072.stderr:512 bytes copied, 0.000186731 s, 2.7 MB/s 2024-03-29T11:34:20.726 DEBUG:teuthology.orchestra.run.smithi072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:34:20.782 DEBUG:teuthology.orchestra.run.smithi072:> stat /dev/vg_nvme/lv_4 2024-03-29T11:34:20.836 INFO:teuthology.orchestra.run.smithi072.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:34:20.836 INFO:teuthology.orchestra.run.smithi072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:34:20.836 INFO:teuthology.orchestra.run.smithi072.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-03-29T11:34:20.836 INFO:teuthology.orchestra.run.smithi072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:34:20.837 INFO:teuthology.orchestra.run.smithi072.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:34:20.837 INFO:teuthology.orchestra.run.smithi072.stdout:Access: 2024-03-29 11:34:15.184883778 +0000 2024-03-29T11:34:20.837 INFO:teuthology.orchestra.run.smithi072.stdout:Modify: 2024-03-29 11:30:18.969165161 +0000 2024-03-29T11:34:20.837 INFO:teuthology.orchestra.run.smithi072.stdout:Change: 2024-03-29 11:30:18.969165161 +0000 2024-03-29T11:34:20.837 INFO:teuthology.orchestra.run.smithi072.stdout: Birth: 2024-03-29 11:30:18.969165161 +0000 2024-03-29T11:34:20.837 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:34:20.902 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records in 2024-03-29T11:34:20.902 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records out 2024-03-29T11:34:20.902 INFO:teuthology.orchestra.run.smithi072.stderr:512 bytes copied, 0.000169367 s, 3.0 MB/s 2024-03-29T11:34:20.903 DEBUG:teuthology.orchestra.run.smithi072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:34:20.959 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:20.959 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-29T11:34:20.959 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:34:20.975 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:34:20.975 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_1 2024-03-29T11:34:21.030 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:34:21.030 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:34:21.031 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-29T11:34:21.031 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:34:21.031 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:34:21.031 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-03-29 11:34:07.499054321 +0000 2024-03-29T11:34:21.031 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-03-29 11:30:04.918414151 +0000 2024-03-29T11:34:21.031 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-03-29 11:30:04.918414151 +0000 2024-03-29T11:34:21.031 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-03-29 11:30:04.918414151 +0000 2024-03-29T11:34:21.031 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:34:21.096 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-03-29T11:34:21.096 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-03-29T11:34:21.096 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000183464 s, 2.8 MB/s 2024-03-29T11:34:21.097 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:34:21.153 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_2 2024-03-29T11:34:21.208 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:34:21.208 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:34:21.208 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-29T11:34:21.208 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:34:21.208 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:34:21.209 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-03-29 11:34:07.500054299 +0000 2024-03-29T11:34:21.209 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-03-29 11:30:05.253406625 +0000 2024-03-29T11:34:21.209 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-03-29 11:30:05.253406625 +0000 2024-03-29T11:34:21.209 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-03-29 11:30:05.253406625 +0000 2024-03-29T11:34:21.209 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:34:21.274 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-03-29T11:34:21.274 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-03-29T11:34:21.274 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.00020383 s, 2.5 MB/s 2024-03-29T11:34:21.275 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:34:21.331 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_3 2024-03-29T11:34:21.386 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:34:21.386 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:34:21.386 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-29T11:34:21.386 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:34:21.386 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:34:21.386 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-03-29 11:34:07.500054299 +0000 2024-03-29T11:34:21.386 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-03-29 11:30:05.602398784 +0000 2024-03-29T11:34:21.386 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-03-29 11:30:05.602398784 +0000 2024-03-29T11:34:21.386 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-03-29 11:30:05.602398784 +0000 2024-03-29T11:34:21.387 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:34:21.452 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-03-29T11:34:21.452 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-03-29T11:34:21.452 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000189224 s, 2.7 MB/s 2024-03-29T11:34:21.453 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:34:21.509 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_4 2024-03-29T11:34:21.564 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:34:21.564 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:34:21.564 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-29T11:34:21.564 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:34:21.564 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:34:21.564 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-03-29 11:34:07.500054299 +0000 2024-03-29T11:34:21.564 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-03-29 11:30:05.950390965 +0000 2024-03-29T11:34:21.564 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-03-29 11:30:05.950390965 +0000 2024-03-29T11:34:21.565 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-03-29 11:30:05.950390965 +0000 2024-03-29T11:34:21.565 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:34:21.629 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-03-29T11:34:21.630 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-03-29T11:34:21.630 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000217715 s, 2.4 MB/s 2024-03-29T11:34:21.631 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:34:21.686 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:21.687 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi072.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi089.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:34:21.687 INFO:tasks.ceph:Generating config... 2024-03-29T11:34:21.688 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T11:34:21.688 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T11:34:21.688 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T11:34:21.688 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T11:34:21.688 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T11:34:21.688 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T11:34:21.688 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T11:34:21.688 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T11:34:21.688 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T11:34:21.689 INFO:tasks.ceph:[global] ms type = async 2024-03-29T11:34:21.689 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T11:34:21.689 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T11:34:21.689 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T11:34:21.689 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T11:34:21.689 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T11:34:21.689 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T11:34:21.689 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T11:34:21.689 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T11:34:21.689 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T11:34:21.689 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T11:34:21.689 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T11:34:21.689 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T11:34:21.689 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T11:34:21.689 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-03-29T11:34:21.689 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T11:34:21.689 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T11:34:21.690 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T11:34:21.690 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T11:34:21.690 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T11:34:21.690 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T11:34:21.690 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T11:34:21.690 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T11:34:21.690 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T11:34:21.690 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T11:34:21.690 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T11:34:21.690 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T11:34:21.690 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T11:34:21.690 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T11:34:21.690 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T11:34:21.690 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T11:34:21.690 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T11:34:21.691 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T11:34:21.691 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T11:34:21.691 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T11:34:21.691 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T11:34:21.691 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T11:34:21.691 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T11:34:21.691 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T11:34:21.691 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T11:34:21.691 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T11:34:21.740 INFO:teuthology.orchestra.run.smithi072.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T11:34:21.743 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T11:34:21.787 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:34:21.814 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.72:6789]'), ('mon.c', '[v1:172.21.15.72:6790]'), ('mon.b', '[v1:172.21.15.89:6789]')] 2024-03-29T11:34:21.814 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.72:6789],[v1:172.21.15.72:6790],[v1:172.21.15.89:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T11:34:21.815 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-29T11:34:21.815 DEBUG:teuthology.orchestra.run.smithi072:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:34:21.868 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.72:6789]' --addv c '[v1:172.21.15.72:6790]' --addv b '[v1:172.21.15.89:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:21.953 INFO:teuthology.orchestra.run.smithi072.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:21.953 INFO:teuthology.orchestra.run.smithi072.stdout:monmaptool: generated fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:34:21.953 INFO:teuthology.orchestra.run.smithi072.stdout:setting min_mon_release = quincy 2024-03-29T11:34:21.953 INFO:teuthology.orchestra.run.smithi072.stdout:epoch 0 2024-03-29T11:34:21.953 INFO:teuthology.orchestra.run.smithi072.stdout:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:34:21.953 INFO:teuthology.orchestra.run.smithi072.stdout:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:34:21.953 INFO:teuthology.orchestra.run.smithi072.stdout:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:34:21.954 INFO:teuthology.orchestra.run.smithi072.stdout:min_mon_release 17 (quincy) 2024-03-29T11:34:21.954 INFO:teuthology.orchestra.run.smithi072.stdout:election_strategy: 1 2024-03-29T11:34:21.954 INFO:teuthology.orchestra.run.smithi072.stdout:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:34:21.954 INFO:teuthology.orchestra.run.smithi072.stdout:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:34:21.954 INFO:teuthology.orchestra.run.smithi072.stdout:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:34:21.954 INFO:teuthology.orchestra.run.smithi072.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T11:34:21.955 DEBUG:teuthology.orchestra.run.smithi072:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:34:21.969 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 5ccd1c04-292b-4e92-9d69-700bf1c298b0... 2024-03-29T11:34:21.970 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:34:22.012 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:34:22.059 INFO:teuthology.orchestra.run.smithi072.stdout:[global] 2024-03-29T11:34:22.059 INFO:teuthology.orchestra.run.smithi072.stdout: chdir = "" 2024-03-29T11:34:22.059 INFO:teuthology.orchestra.run.smithi072.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:34:22.060 INFO:teuthology.orchestra.run.smithi072.stdout: auth supported = cephx 2024-03-29T11:34:22.060 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:22.060 INFO:teuthology.orchestra.run.smithi072.stdout: filestore xattr use omap = true 2024-03-29T11:34:22.060 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:22.060 INFO:teuthology.orchestra.run.smithi072.stdout: mon clock drift allowed = 1.000 2024-03-29T11:34:22.060 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:22.060 INFO:teuthology.orchestra.run.smithi072.stdout: osd crush chooseleaf type = 0 2024-03-29T11:34:22.060 INFO:teuthology.orchestra.run.smithi072.stdout: auth debug = true 2024-03-29T11:34:22.060 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:22.060 INFO:teuthology.orchestra.run.smithi072.stdout: ms die on old message = true 2024-03-29T11:34:22.060 INFO:teuthology.orchestra.run.smithi072.stdout: ms die on bug = true 2024-03-29T11:34:22.060 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:22.060 INFO:teuthology.orchestra.run.smithi072.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:34:22.060 INFO:teuthology.orchestra.run.smithi072.stdout: mon pg warn max object skew = 0 2024-03-29T11:34:22.060 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:22.061 INFO:teuthology.orchestra.run.smithi072.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:34:22.061 INFO:teuthology.orchestra.run.smithi072.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:34:22.061 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:22.061 INFO:teuthology.orchestra.run.smithi072.stdout: osd pool default size = 2 2024-03-29T11:34:22.061 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:22.061 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd allow primary affinity = true 2024-03-29T11:34:22.061 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd allow pg remap = true 2024-03-29T11:34:22.061 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:34:22.061 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:34:22.061 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on no sortbitwise = false 2024-03-29T11:34:22.061 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:34:22.061 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on too few osds = false 2024-03-29T11:34:22.061 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:34:22.061 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:34:22.061 INFO:teuthology.orchestra.run.smithi072.stdout: mon_allow_pool_size_one = true 2024-03-29T11:34:22.061 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:22.062 INFO:teuthology.orchestra.run.smithi072.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:34:22.062 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:22.062 INFO:teuthology.orchestra.run.smithi072.stdout: osd default data pool replay window = 5 2024-03-29T11:34:22.062 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:22.062 INFO:teuthology.orchestra.run.smithi072.stdout: mon allow pool delete = true 2024-03-29T11:34:22.062 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:22.062 INFO:teuthology.orchestra.run.smithi072.stdout: mon cluster log file level = debug 2024-03-29T11:34:22.062 INFO:teuthology.orchestra.run.smithi072.stdout: debug asserts on shutdown = true 2024-03-29T11:34:22.062 INFO:teuthology.orchestra.run.smithi072.stdout: mon health detail to clog = false 2024-03-29T11:34:22.062 INFO:teuthology.orchestra.run.smithi072.stdout: mon host = "[v1:172.21.15.72:6789],[v1:172.21.15.72:6790],[v1:172.21.15.89:6789]" 2024-03-29T11:34:22.062 INFO:teuthology.orchestra.run.smithi072.stdout: lockdep = True 2024-03-29T11:34:22.062 INFO:teuthology.orchestra.run.smithi072.stdout: mon client directed command retry = 5 2024-03-29T11:34:22.062 INFO:teuthology.orchestra.run.smithi072.stdout: mon election default strategy = 1 2024-03-29T11:34:22.062 INFO:teuthology.orchestra.run.smithi072.stdout: ms bind msgr2 = False 2024-03-29T11:34:22.063 INFO:teuthology.orchestra.run.smithi072.stdout: ms inject socket failures = 5000 2024-03-29T11:34:22.063 INFO:teuthology.orchestra.run.smithi072.stdout: ms type = async 2024-03-29T11:34:22.063 INFO:teuthology.orchestra.run.smithi072.stdout: fsid = 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:34:22.063 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:22.063 INFO:teuthology.orchestra.run.smithi072.stdout:[osd] 2024-03-29T11:34:22.063 INFO:teuthology.orchestra.run.smithi072.stdout: osd journal size = 100 2024-03-29T11:34:22.063 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:22.063 INFO:teuthology.orchestra.run.smithi072.stdout: osd scrub load threshold = 5.0 2024-03-29T11:34:22.063 INFO:teuthology.orchestra.run.smithi072.stdout: osd scrub max interval = 600 2024-03-29T11:34:22.063 INFO:teuthology.orchestra.run.smithi072.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:34:22.063 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:22.063 INFO:teuthology.orchestra.run.smithi072.stdout: osd recover clone overlap = true 2024-03-29T11:34:22.063 INFO:teuthology.orchestra.run.smithi072.stdout: osd recovery max chunk = 1048576 2024-03-29T11:34:22.063 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:22.063 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug shutdown = true 2024-03-29T11:34:22.063 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug op order = true 2024-03-29T11:34:22.064 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug verify stray on activate = true 2024-03-29T11:34:22.064 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug trim objects = true 2024-03-29T11:34:22.064 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:22.064 INFO:teuthology.orchestra.run.smithi072.stdout: osd open classes on start = true 2024-03-29T11:34:22.064 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug pg log writeout = true 2024-03-29T11:34:22.064 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:22.064 INFO:teuthology.orchestra.run.smithi072.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:34:22.064 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:22.064 INFO:teuthology.orchestra.run.smithi072.stdout: osd map max advance = 10 2024-03-29T11:34:22.064 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:22.064 INFO:teuthology.orchestra.run.smithi072.stdout: journal zero on create = true 2024-03-29T11:34:22.064 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:22.064 INFO:teuthology.orchestra.run.smithi072.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:34:22.064 INFO:teuthology.orchestra.run.smithi072.stdout: filestore apply finisher threads = 3 2024-03-29T11:34:22.064 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:22.065 INFO:teuthology.orchestra.run.smithi072.stdout: bdev debug aio = true 2024-03-29T11:34:22.065 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug misdirected ops = true 2024-03-29T11:34:22.065 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore block size = 96636764160 2024-03-29T11:34:22.065 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore compression algorithm = snappy 2024-03-29T11:34:22.065 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore compression mode = aggressive 2024-03-29T11:34:22.065 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore fsck on mount = True 2024-03-29T11:34:22.065 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore zero block detection = True 2024-03-29T11:34:22.065 INFO:teuthology.orchestra.run.smithi072.stdout: debug bluefs = 20 2024-03-29T11:34:22.065 INFO:teuthology.orchestra.run.smithi072.stdout: debug bluestore = 20 2024-03-29T11:34:22.065 INFO:teuthology.orchestra.run.smithi072.stdout: debug monc = 20 2024-03-29T11:34:22.065 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-29T11:34:22.065 INFO:teuthology.orchestra.run.smithi072.stdout: debug osd = 20 2024-03-29T11:34:22.065 INFO:teuthology.orchestra.run.smithi072.stdout: debug rocksdb = 10 2024-03-29T11:34:22.065 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:34:22.065 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd full ratio = 0.9 2024-03-29T11:34:22.066 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:34:22.066 INFO:teuthology.orchestra.run.smithi072.stdout: osd class default list = * 2024-03-29T11:34:22.066 INFO:teuthology.orchestra.run.smithi072.stdout: osd class load list = * 2024-03-29T11:34:22.066 INFO:teuthology.orchestra.run.smithi072.stdout: osd client watch timeout = 120 2024-03-29T11:34:22.066 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug verify cached snaps = True 2024-03-29T11:34:22.066 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug verify missing on start = True 2024-03-29T11:34:22.066 INFO:teuthology.orchestra.run.smithi072.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:34:22.066 INFO:teuthology.orchestra.run.smithi072.stdout: osd mclock override recovery settings = True 2024-03-29T11:34:22.066 INFO:teuthology.orchestra.run.smithi072.stdout: osd objectstore = bluestore 2024-03-29T11:34:22.066 INFO:teuthology.orchestra.run.smithi072.stdout: osd op queue = debug_random 2024-03-29T11:34:22.066 INFO:teuthology.orchestra.run.smithi072.stdout: osd op queue cut off = debug_random 2024-03-29T11:34:22.066 INFO:teuthology.orchestra.run.smithi072.stdout: osd shutdown pgref assert = True 2024-03-29T11:34:22.066 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:22.066 INFO:teuthology.orchestra.run.smithi072.stdout:[mgr] 2024-03-29T11:34:22.066 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-29T11:34:22.066 INFO:teuthology.orchestra.run.smithi072.stdout: debug mgr = 20 2024-03-29T11:34:22.067 INFO:teuthology.orchestra.run.smithi072.stdout: debug mon = 20 2024-03-29T11:34:22.067 INFO:teuthology.orchestra.run.smithi072.stdout: debug auth = 20 2024-03-29T11:34:22.067 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:34:22.067 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:34:22.067 INFO:teuthology.orchestra.run.smithi072.stdout: mgr/telemetry/nag = false 2024-03-29T11:34:22.067 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:22.067 INFO:teuthology.orchestra.run.smithi072.stdout:[mon] 2024-03-29T11:34:22.067 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-29T11:34:22.067 INFO:teuthology.orchestra.run.smithi072.stdout: debug mon = 20 2024-03-29T11:34:22.067 INFO:teuthology.orchestra.run.smithi072.stdout: debug paxos = 20 2024-03-29T11:34:22.067 INFO:teuthology.orchestra.run.smithi072.stdout: debug auth = 20 2024-03-29T11:34:22.067 INFO:teuthology.orchestra.run.smithi072.stdout: mon data avail warn = 5 2024-03-29T11:34:22.067 INFO:teuthology.orchestra.run.smithi072.stdout: mon mgr mkfs grace = 240 2024-03-29T11:34:22.067 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:34:22.067 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd reporter subtree level = osd 2024-03-29T11:34:22.067 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd prime pg temp = true 2024-03-29T11:34:22.068 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:34:22.068 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:22.068 INFO:teuthology.orchestra.run.smithi072.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:34:22.068 INFO:teuthology.orchestra.run.smithi072.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:34:22.068 INFO:teuthology.orchestra.run.smithi072.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:34:22.068 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:22.068 INFO:teuthology.orchestra.run.smithi072.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:34:22.068 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:34:22.068 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:34:22.068 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:22.068 INFO:teuthology.orchestra.run.smithi072.stdout: # 1m isn't quite enough 2024-03-29T11:34:22.068 INFO:teuthology.orchestra.run.smithi072.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:34:22.068 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:22.068 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:34:22.068 INFO:teuthology.orchestra.run.smithi072.stdout: mon min osdmap epochs = 50 2024-03-29T11:34:22.068 INFO:teuthology.orchestra.run.smithi072.stdout: mon osdmap full prune interval = 2 2024-03-29T11:34:22.069 INFO:teuthology.orchestra.run.smithi072.stdout: mon osdmap full prune min = 15 2024-03-29T11:34:22.069 INFO:teuthology.orchestra.run.smithi072.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:34:22.069 INFO:teuthology.orchestra.run.smithi072.stdout: mon scrub interval = 300 2024-03-29T11:34:22.069 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on pool no app = False 2024-03-29T11:34:22.069 INFO:teuthology.orchestra.run.smithi072.stdout: paxos service trim min = 10 2024-03-29T11:34:22.069 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:34:22.069 INFO:teuthology.orchestra.run.smithi072.stdout:[client] 2024-03-29T11:34:22.069 INFO:teuthology.orchestra.run.smithi072.stdout: rgw cache enabled = true 2024-03-29T11:34:22.069 INFO:teuthology.orchestra.run.smithi072.stdout: rgw enable ops log = true 2024-03-29T11:34:22.069 INFO:teuthology.orchestra.run.smithi072.stdout: rgw enable usage log = true 2024-03-29T11:34:22.069 INFO:teuthology.orchestra.run.smithi072.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:34:22.069 INFO:teuthology.orchestra.run.smithi072.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:34:22.069 INFO:teuthology.orchestra.run.smithi072.stdout: debug monc = 20 2024-03-29T11:34:22.069 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-03-29T11:34:22.069 INFO:teuthology.orchestra.run.smithi072.stdout: debug objecter = 20 2024-03-29T11:34:22.070 INFO:teuthology.orchestra.run.smithi072.stdout: debug rados = 20 2024-03-29T11:34:22.070 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.a] 2024-03-29T11:34:22.070 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.c] 2024-03-29T11:34:22.070 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.b] 2024-03-29T11:34:22.070 INFO:teuthology.orchestra.run.smithi089.stdout:[global] 2024-03-29T11:34:22.070 INFO:teuthology.orchestra.run.smithi089.stdout: chdir = "" 2024-03-29T11:34:22.070 INFO:teuthology.orchestra.run.smithi089.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:34:22.070 INFO:teuthology.orchestra.run.smithi089.stdout: auth supported = cephx 2024-03-29T11:34:22.070 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:34:22.071 INFO:teuthology.orchestra.run.smithi089.stdout: filestore xattr use omap = true 2024-03-29T11:34:22.071 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:34:22.071 INFO:teuthology.orchestra.run.smithi089.stdout: mon clock drift allowed = 1.000 2024-03-29T11:34:22.071 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:34:22.071 INFO:teuthology.orchestra.run.smithi089.stdout: osd crush chooseleaf type = 0 2024-03-29T11:34:22.071 INFO:teuthology.orchestra.run.smithi089.stdout: auth debug = true 2024-03-29T11:34:22.071 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:34:22.071 INFO:teuthology.orchestra.run.smithi089.stdout: ms die on old message = true 2024-03-29T11:34:22.071 INFO:teuthology.orchestra.run.smithi089.stdout: ms die on bug = true 2024-03-29T11:34:22.071 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:34:22.071 INFO:teuthology.orchestra.run.smithi089.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:34:22.071 INFO:teuthology.orchestra.run.smithi089.stdout: mon pg warn max object skew = 0 2024-03-29T11:34:22.071 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:34:22.071 INFO:teuthology.orchestra.run.smithi089.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:34:22.071 INFO:teuthology.orchestra.run.smithi089.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:34:22.072 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:34:22.072 INFO:teuthology.orchestra.run.smithi089.stdout: osd pool default size = 2 2024-03-29T11:34:22.072 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:34:22.072 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd allow primary affinity = true 2024-03-29T11:34:22.072 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd allow pg remap = true 2024-03-29T11:34:22.072 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:34:22.072 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:34:22.072 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on no sortbitwise = false 2024-03-29T11:34:22.072 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:34:22.072 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on too few osds = false 2024-03-29T11:34:22.072 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:34:22.072 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:34:22.072 INFO:teuthology.orchestra.run.smithi089.stdout: mon_allow_pool_size_one = true 2024-03-29T11:34:22.072 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:34:22.072 INFO:teuthology.orchestra.run.smithi089.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:34:22.072 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:34:22.073 INFO:teuthology.orchestra.run.smithi089.stdout: osd default data pool replay window = 5 2024-03-29T11:34:22.073 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:34:22.073 INFO:teuthology.orchestra.run.smithi089.stdout: mon allow pool delete = true 2024-03-29T11:34:22.073 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:34:22.073 INFO:teuthology.orchestra.run.smithi089.stdout: mon cluster log file level = debug 2024-03-29T11:34:22.073 INFO:teuthology.orchestra.run.smithi089.stdout: debug asserts on shutdown = true 2024-03-29T11:34:22.073 INFO:teuthology.orchestra.run.smithi089.stdout: mon health detail to clog = false 2024-03-29T11:34:22.073 INFO:teuthology.orchestra.run.smithi089.stdout: mon host = "[v1:172.21.15.72:6789],[v1:172.21.15.72:6790],[v1:172.21.15.89:6789]" 2024-03-29T11:34:22.073 INFO:teuthology.orchestra.run.smithi089.stdout: lockdep = True 2024-03-29T11:34:22.073 INFO:teuthology.orchestra.run.smithi089.stdout: mon client directed command retry = 5 2024-03-29T11:34:22.073 INFO:teuthology.orchestra.run.smithi089.stdout: mon election default strategy = 1 2024-03-29T11:34:22.073 INFO:teuthology.orchestra.run.smithi089.stdout: ms bind msgr2 = False 2024-03-29T11:34:22.073 INFO:teuthology.orchestra.run.smithi089.stdout: ms inject socket failures = 5000 2024-03-29T11:34:22.073 INFO:teuthology.orchestra.run.smithi089.stdout: ms type = async 2024-03-29T11:34:22.073 INFO:teuthology.orchestra.run.smithi089.stdout: fsid = 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:34:22.073 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:34:22.074 INFO:teuthology.orchestra.run.smithi089.stdout:[osd] 2024-03-29T11:34:22.074 INFO:teuthology.orchestra.run.smithi089.stdout: osd journal size = 100 2024-03-29T11:34:22.074 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:34:22.074 INFO:teuthology.orchestra.run.smithi089.stdout: osd scrub load threshold = 5.0 2024-03-29T11:34:22.074 INFO:teuthology.orchestra.run.smithi089.stdout: osd scrub max interval = 600 2024-03-29T11:34:22.074 INFO:teuthology.orchestra.run.smithi089.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:34:22.074 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:34:22.074 INFO:teuthology.orchestra.run.smithi089.stdout: osd recover clone overlap = true 2024-03-29T11:34:22.074 INFO:teuthology.orchestra.run.smithi089.stdout: osd recovery max chunk = 1048576 2024-03-29T11:34:22.074 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:34:22.074 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug shutdown = true 2024-03-29T11:34:22.074 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug op order = true 2024-03-29T11:34:22.074 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug verify stray on activate = true 2024-03-29T11:34:22.074 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug trim objects = true 2024-03-29T11:34:22.074 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:34:22.075 INFO:teuthology.orchestra.run.smithi089.stdout: osd open classes on start = true 2024-03-29T11:34:22.075 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug pg log writeout = true 2024-03-29T11:34:22.075 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:34:22.075 INFO:teuthology.orchestra.run.smithi089.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:34:22.075 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:34:22.075 INFO:teuthology.orchestra.run.smithi089.stdout: osd map max advance = 10 2024-03-29T11:34:22.075 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:34:22.075 INFO:teuthology.orchestra.run.smithi089.stdout: journal zero on create = true 2024-03-29T11:34:22.075 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:34:22.075 INFO:teuthology.orchestra.run.smithi089.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:34:22.075 INFO:teuthology.orchestra.run.smithi089.stdout: filestore apply finisher threads = 3 2024-03-29T11:34:22.075 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:34:22.075 INFO:teuthology.orchestra.run.smithi089.stdout: bdev debug aio = true 2024-03-29T11:34:22.075 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug misdirected ops = true 2024-03-29T11:34:22.075 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore block size = 96636764160 2024-03-29T11:34:22.076 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore compression algorithm = snappy 2024-03-29T11:34:22.076 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore compression mode = aggressive 2024-03-29T11:34:22.076 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore fsck on mount = True 2024-03-29T11:34:22.076 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore zero block detection = True 2024-03-29T11:34:22.076 INFO:teuthology.orchestra.run.smithi089.stdout: debug bluefs = 20 2024-03-29T11:34:22.076 INFO:teuthology.orchestra.run.smithi089.stdout: debug bluestore = 20 2024-03-29T11:34:22.076 INFO:teuthology.orchestra.run.smithi089.stdout: debug monc = 20 2024-03-29T11:34:22.076 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-03-29T11:34:22.076 INFO:teuthology.orchestra.run.smithi089.stdout: debug osd = 20 2024-03-29T11:34:22.076 INFO:teuthology.orchestra.run.smithi089.stdout: debug rocksdb = 10 2024-03-29T11:34:22.076 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:34:22.076 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd full ratio = 0.9 2024-03-29T11:34:22.076 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:34:22.076 INFO:teuthology.orchestra.run.smithi089.stdout: osd class default list = * 2024-03-29T11:34:22.076 INFO:teuthology.orchestra.run.smithi089.stdout: osd class load list = * 2024-03-29T11:34:22.076 INFO:teuthology.orchestra.run.smithi089.stdout: osd client watch timeout = 120 2024-03-29T11:34:22.077 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug verify cached snaps = True 2024-03-29T11:34:22.077 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug verify missing on start = True 2024-03-29T11:34:22.077 INFO:teuthology.orchestra.run.smithi089.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:34:22.077 INFO:teuthology.orchestra.run.smithi089.stdout: osd mclock override recovery settings = True 2024-03-29T11:34:22.077 INFO:teuthology.orchestra.run.smithi089.stdout: osd objectstore = bluestore 2024-03-29T11:34:22.077 INFO:teuthology.orchestra.run.smithi089.stdout: osd op queue = debug_random 2024-03-29T11:34:22.077 INFO:teuthology.orchestra.run.smithi089.stdout: osd op queue cut off = debug_random 2024-03-29T11:34:22.077 INFO:teuthology.orchestra.run.smithi089.stdout: osd shutdown pgref assert = True 2024-03-29T11:34:22.077 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:34:22.077 INFO:teuthology.orchestra.run.smithi089.stdout:[mgr] 2024-03-29T11:34:22.077 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-03-29T11:34:22.077 INFO:teuthology.orchestra.run.smithi089.stdout: debug mgr = 20 2024-03-29T11:34:22.077 INFO:teuthology.orchestra.run.smithi089.stdout: debug mon = 20 2024-03-29T11:34:22.077 INFO:teuthology.orchestra.run.smithi089.stdout: debug auth = 20 2024-03-29T11:34:22.077 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:34:22.078 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:34:22.078 INFO:teuthology.orchestra.run.smithi089.stdout: mgr/telemetry/nag = false 2024-03-29T11:34:22.078 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:34:22.078 INFO:teuthology.orchestra.run.smithi089.stdout:[mon] 2024-03-29T11:34:22.078 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-03-29T11:34:22.078 INFO:teuthology.orchestra.run.smithi089.stdout: debug mon = 20 2024-03-29T11:34:22.078 INFO:teuthology.orchestra.run.smithi089.stdout: debug paxos = 20 2024-03-29T11:34:22.078 INFO:teuthology.orchestra.run.smithi089.stdout: debug auth = 20 2024-03-29T11:34:22.078 INFO:teuthology.orchestra.run.smithi089.stdout: mon data avail warn = 5 2024-03-29T11:34:22.078 INFO:teuthology.orchestra.run.smithi089.stdout: mon mgr mkfs grace = 240 2024-03-29T11:34:22.078 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:34:22.078 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd reporter subtree level = osd 2024-03-29T11:34:22.078 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd prime pg temp = true 2024-03-29T11:34:22.078 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:34:22.078 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:34:22.078 INFO:teuthology.orchestra.run.smithi089.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:34:22.079 INFO:teuthology.orchestra.run.smithi089.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:34:22.079 INFO:teuthology.orchestra.run.smithi089.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:34:22.079 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:34:22.079 INFO:teuthology.orchestra.run.smithi089.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:34:22.079 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:34:22.079 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:34:22.079 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:34:22.079 INFO:teuthology.orchestra.run.smithi089.stdout: # 1m isn't quite enough 2024-03-29T11:34:22.079 INFO:teuthology.orchestra.run.smithi089.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:34:22.079 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:34:22.079 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:34:22.079 INFO:teuthology.orchestra.run.smithi089.stdout: mon min osdmap epochs = 50 2024-03-29T11:34:22.079 INFO:teuthology.orchestra.run.smithi089.stdout: mon osdmap full prune interval = 2 2024-03-29T11:34:22.079 INFO:teuthology.orchestra.run.smithi089.stdout: mon osdmap full prune min = 15 2024-03-29T11:34:22.079 INFO:teuthology.orchestra.run.smithi089.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:34:22.080 INFO:teuthology.orchestra.run.smithi089.stdout: mon scrub interval = 300 2024-03-29T11:34:22.080 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on pool no app = False 2024-03-29T11:34:22.080 INFO:teuthology.orchestra.run.smithi089.stdout: paxos service trim min = 10 2024-03-29T11:34:22.080 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-03-29T11:34:22.080 INFO:teuthology.orchestra.run.smithi089.stdout:[client] 2024-03-29T11:34:22.080 INFO:teuthology.orchestra.run.smithi089.stdout: rgw cache enabled = true 2024-03-29T11:34:22.080 INFO:teuthology.orchestra.run.smithi089.stdout: rgw enable ops log = true 2024-03-29T11:34:22.080 INFO:teuthology.orchestra.run.smithi089.stdout: rgw enable usage log = true 2024-03-29T11:34:22.080 INFO:teuthology.orchestra.run.smithi089.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:34:22.080 INFO:teuthology.orchestra.run.smithi089.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:34:22.080 INFO:teuthology.orchestra.run.smithi089.stdout: debug monc = 20 2024-03-29T11:34:22.080 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-03-29T11:34:22.080 INFO:teuthology.orchestra.run.smithi089.stdout: debug objecter = 20 2024-03-29T11:34:22.080 INFO:teuthology.orchestra.run.smithi089.stdout: debug rados = 20 2024-03-29T11:34:22.080 INFO:teuthology.orchestra.run.smithi089.stdout:[mon.a] 2024-03-29T11:34:22.080 INFO:teuthology.orchestra.run.smithi089.stdout:[mon.c] 2024-03-29T11:34:22.081 INFO:teuthology.orchestra.run.smithi089.stdout:[mon.b] 2024-03-29T11:34:22.082 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T11:34:22.082 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T11:34:22.177 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T11:34:22.177 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-29T11:34:22.177 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T11:34:22.192 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-29T11:34:22.192 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T11:34:22.247 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi072.front.sepia.ceph.com 2024-03-29T11:34:22.247 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-29T11:34:22.247 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:34:22.247 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:34:22.326 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-29T11:34:22.326 DEBUG:teuthology.orchestra.run.smithi072:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:22.380 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi089.front.sepia.ceph.com 2024-03-29T11:34:22.381 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-29T11:34:22.381 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:34:22.381 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:34:22.421 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-29T11:34:22.421 DEBUG:teuthology.orchestra.run.smithi089:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:22.476 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T11:34:22.476 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T11:34:22.476 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:34:22.531 INFO:teuthology.orchestra.run.smithi072.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:34:22.533 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:34:22.604 INFO:teuthology.orchestra.run.smithi089.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:34:22.606 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T11:34:22.607 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T11:34:22.607 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T11:34:22.649 INFO:teuthology.orchestra.run.smithi072.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T11:34:22.664 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T11:34:22.706 INFO:teuthology.orchestra.run.smithi089.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T11:34:22.723 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T11:34:22.723 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi072.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi089.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:34:22.723 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T11:34:22.750 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:22.750 INFO:tasks.ceph:role: osd.0 2024-03-29T11:34:22.750 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi072.front.sepia.ceph.com 2024-03-29T11:34:22.750 DEBUG:teuthology.orchestra.run.smithi072:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:34:22.824 INFO:teuthology.orchestra.run.smithi072.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:34:22.824 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:34:22.824 INFO:teuthology.orchestra.run.smithi072.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:34:22.825 INFO:teuthology.orchestra.run.smithi072.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:34:22.825 INFO:teuthology.orchestra.run.smithi072.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:34:22.825 INFO:teuthology.orchestra.run.smithi072.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:34:22.825 INFO:teuthology.orchestra.run.smithi072.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:34:22.825 INFO:teuthology.orchestra.run.smithi072.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:34:22.825 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:34:22.826 INFO:teuthology.orchestra.run.smithi072.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:34:23.325 INFO:teuthology.orchestra.run.smithi072.stdout:Discarding blocks...Done. 2024-03-29T11:34:23.327 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi072.front.sepia.ceph.com -o noatime 2024-03-29T11:34:23.327 DEBUG:teuthology.orchestra.run.smithi072:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T11:34:23.459 DEBUG:teuthology.orchestra.run.smithi072:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T11:34:23.494 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T11:34:23.561 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:23.561 INFO:tasks.ceph:role: osd.1 2024-03-29T11:34:23.561 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi072.front.sepia.ceph.com 2024-03-29T11:34:23.561 DEBUG:teuthology.orchestra.run.smithi072:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:34:23.636 INFO:teuthology.orchestra.run.smithi072.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:34:23.636 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:34:23.636 INFO:teuthology.orchestra.run.smithi072.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:34:23.636 INFO:teuthology.orchestra.run.smithi072.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:34:23.636 INFO:teuthology.orchestra.run.smithi072.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:34:23.636 INFO:teuthology.orchestra.run.smithi072.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:34:23.636 INFO:teuthology.orchestra.run.smithi072.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:34:23.636 INFO:teuthology.orchestra.run.smithi072.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:34:23.636 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:34:23.636 INFO:teuthology.orchestra.run.smithi072.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:34:24.092 INFO:teuthology.orchestra.run.smithi072.stdout:Discarding blocks...Done. 2024-03-29T11:34:24.095 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi072.front.sepia.ceph.com -o noatime 2024-03-29T11:34:24.095 DEBUG:teuthology.orchestra.run.smithi072:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T11:34:24.191 DEBUG:teuthology.orchestra.run.smithi072:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T11:34:24.223 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T11:34:24.291 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:24.291 INFO:tasks.ceph:role: osd.2 2024-03-29T11:34:24.291 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi072.front.sepia.ceph.com 2024-03-29T11:34:24.291 DEBUG:teuthology.orchestra.run.smithi072:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:34:24.365 INFO:teuthology.orchestra.run.smithi072.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:34:24.365 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:34:24.365 INFO:teuthology.orchestra.run.smithi072.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:34:24.365 INFO:teuthology.orchestra.run.smithi072.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:34:24.365 INFO:teuthology.orchestra.run.smithi072.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:34:24.365 INFO:teuthology.orchestra.run.smithi072.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:34:24.365 INFO:teuthology.orchestra.run.smithi072.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:34:24.365 INFO:teuthology.orchestra.run.smithi072.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:34:24.365 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:34:24.365 INFO:teuthology.orchestra.run.smithi072.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:34:24.857 INFO:teuthology.orchestra.run.smithi072.stdout:Discarding blocks...Done. 2024-03-29T11:34:24.859 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi072.front.sepia.ceph.com -o noatime 2024-03-29T11:34:24.859 DEBUG:teuthology.orchestra.run.smithi072:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T11:34:25.000 DEBUG:teuthology.orchestra.run.smithi072:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T11:34:25.031 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T11:34:25.098 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:25.099 INFO:tasks.ceph:role: osd.3 2024-03-29T11:34:25.099 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi072.front.sepia.ceph.com 2024-03-29T11:34:25.099 DEBUG:teuthology.orchestra.run.smithi072:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:34:25.174 INFO:teuthology.orchestra.run.smithi072.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:34:25.175 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:34:25.175 INFO:teuthology.orchestra.run.smithi072.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:34:25.175 INFO:teuthology.orchestra.run.smithi072.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:34:25.175 INFO:teuthology.orchestra.run.smithi072.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:34:25.175 INFO:teuthology.orchestra.run.smithi072.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:34:25.175 INFO:teuthology.orchestra.run.smithi072.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:34:25.175 INFO:teuthology.orchestra.run.smithi072.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:34:25.175 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:34:25.175 INFO:teuthology.orchestra.run.smithi072.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:34:25.685 INFO:teuthology.orchestra.run.smithi072.stdout:Discarding blocks...Done. 2024-03-29T11:34:25.687 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi072.front.sepia.ceph.com -o noatime 2024-03-29T11:34:25.687 DEBUG:teuthology.orchestra.run.smithi072:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T11:34:25.826 DEBUG:teuthology.orchestra.run.smithi072:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T11:34:25.858 DEBUG:teuthology.orchestra.run.smithi072:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:25.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:25.961+0000 7efebb066640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T11:34:25.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:25.961+0000 7efebb066640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T11:34:25.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:25.961+0000 7efebb066640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:34:25.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:25.961+0000 7efebb066640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:34:25.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:25.961+0000 7efebb066640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:34:25.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:25.962+0000 7efebb066640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T11:34:28.806 DEBUG:teuthology.orchestra.run.smithi072:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T11:34:28.837 DEBUG:teuthology.orchestra.run.smithi072:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:28.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:28.931+0000 7feb5cf16640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T11:34:28.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:28.931+0000 7feb5cf16640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T11:34:28.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:28.931+0000 7feb5cf16640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:34:28.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:28.931+0000 7feb5cf16640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:34:28.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:28.931+0000 7feb5cf16640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:34:28.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:28.932+0000 7feb5cf16640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T11:34:31.771 DEBUG:teuthology.orchestra.run.smithi072:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T11:34:31.801 DEBUG:teuthology.orchestra.run.smithi072:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:31.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:31.892+0000 7f7cdb945640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T11:34:31.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:31.893+0000 7f7cdb945640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T11:34:31.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:31.893+0000 7f7cdb945640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:34:31.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:31.893+0000 7f7cdb945640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:34:31.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:31.893+0000 7f7cdb945640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:34:31.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:31.893+0000 7f7cdb945640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T11:34:34.751 DEBUG:teuthology.orchestra.run.smithi072:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T11:34:34.781 DEBUG:teuthology.orchestra.run.smithi072:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:34.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:34.873+0000 7fbdc70d9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T11:34:34.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:34.873+0000 7fbdc70d9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T11:34:34.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:34.873+0000 7fbdc70d9640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:34:34.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:34.873+0000 7fbdc70d9640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:34:34.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:34.873+0000 7fbdc70d9640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:34:34.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:34.874+0000 7fbdc70d9640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T11:34:37.720 DEBUG:teuthology.orchestra.run.smithi072:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T11:34:37.751 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T11:34:37.783 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:37.784 INFO:tasks.ceph:role: osd.4 2024-03-29T11:34:37.784 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi089.front.sepia.ceph.com 2024-03-29T11:34:37.784 DEBUG:teuthology.orchestra.run.smithi089:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:34:37.861 INFO:teuthology.orchestra.run.smithi089.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:34:37.861 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:34:37.861 INFO:teuthology.orchestra.run.smithi089.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:34:37.861 INFO:teuthology.orchestra.run.smithi089.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:34:37.861 INFO:teuthology.orchestra.run.smithi089.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:34:37.861 INFO:teuthology.orchestra.run.smithi089.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:34:37.861 INFO:teuthology.orchestra.run.smithi089.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:34:37.861 INFO:teuthology.orchestra.run.smithi089.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:34:37.861 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:34:37.862 INFO:teuthology.orchestra.run.smithi089.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:34:38.382 INFO:teuthology.orchestra.run.smithi089.stdout:Discarding blocks...Done. 2024-03-29T11:34:38.384 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi089.front.sepia.ceph.com -o noatime 2024-03-29T11:34:38.384 DEBUG:teuthology.orchestra.run.smithi089:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T11:34:38.525 DEBUG:teuthology.orchestra.run.smithi089:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T11:34:38.562 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T11:34:38.632 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:38.633 INFO:tasks.ceph:role: osd.5 2024-03-29T11:34:38.633 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi089.front.sepia.ceph.com 2024-03-29T11:34:38.633 DEBUG:teuthology.orchestra.run.smithi089:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:34:38.708 INFO:teuthology.orchestra.run.smithi089.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:34:38.709 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:34:38.709 INFO:teuthology.orchestra.run.smithi089.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:34:38.709 INFO:teuthology.orchestra.run.smithi089.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:34:38.709 INFO:teuthology.orchestra.run.smithi089.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:34:38.709 INFO:teuthology.orchestra.run.smithi089.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:34:38.709 INFO:teuthology.orchestra.run.smithi089.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:34:38.709 INFO:teuthology.orchestra.run.smithi089.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:34:38.709 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:34:38.709 INFO:teuthology.orchestra.run.smithi089.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:34:39.153 INFO:teuthology.orchestra.run.smithi089.stdout:Discarding blocks...Done. 2024-03-29T11:34:39.155 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi089.front.sepia.ceph.com -o noatime 2024-03-29T11:34:39.155 DEBUG:teuthology.orchestra.run.smithi089:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T11:34:39.256 DEBUG:teuthology.orchestra.run.smithi089:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T11:34:39.289 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T11:34:39.356 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:39.356 INFO:tasks.ceph:role: osd.6 2024-03-29T11:34:39.356 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi089.front.sepia.ceph.com 2024-03-29T11:34:39.356 DEBUG:teuthology.orchestra.run.smithi089:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:34:39.430 INFO:teuthology.orchestra.run.smithi089.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:34:39.430 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:34:39.430 INFO:teuthology.orchestra.run.smithi089.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:34:39.430 INFO:teuthology.orchestra.run.smithi089.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:34:39.430 INFO:teuthology.orchestra.run.smithi089.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:34:39.431 INFO:teuthology.orchestra.run.smithi089.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:34:39.431 INFO:teuthology.orchestra.run.smithi089.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:34:39.431 INFO:teuthology.orchestra.run.smithi089.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:34:39.431 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:34:39.431 INFO:teuthology.orchestra.run.smithi089.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:34:39.912 INFO:teuthology.orchestra.run.smithi089.stdout:Discarding blocks...Done. 2024-03-29T11:34:39.913 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi089.front.sepia.ceph.com -o noatime 2024-03-29T11:34:39.914 DEBUG:teuthology.orchestra.run.smithi089:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T11:34:40.047 DEBUG:teuthology.orchestra.run.smithi089:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T11:34:40.079 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T11:34:40.146 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:40.147 INFO:tasks.ceph:role: osd.7 2024-03-29T11:34:40.147 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi089.front.sepia.ceph.com 2024-03-29T11:34:40.147 DEBUG:teuthology.orchestra.run.smithi089:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:34:40.220 INFO:teuthology.orchestra.run.smithi089.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:34:40.220 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:34:40.220 INFO:teuthology.orchestra.run.smithi089.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:34:40.221 INFO:teuthology.orchestra.run.smithi089.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:34:40.221 INFO:teuthology.orchestra.run.smithi089.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:34:40.221 INFO:teuthology.orchestra.run.smithi089.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:34:40.221 INFO:teuthology.orchestra.run.smithi089.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:34:40.221 INFO:teuthology.orchestra.run.smithi089.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:34:40.221 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:34:40.221 INFO:teuthology.orchestra.run.smithi089.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:34:40.732 INFO:teuthology.orchestra.run.smithi089.stdout:Discarding blocks...Done. 2024-03-29T11:34:40.734 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi089.front.sepia.ceph.com -o noatime 2024-03-29T11:34:40.734 DEBUG:teuthology.orchestra.run.smithi089:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T11:34:40.854 DEBUG:teuthology.orchestra.run.smithi089:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T11:34:40.888 DEBUG:teuthology.orchestra.run.smithi089:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:40.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:34:41.000+0000 7f5f68462640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T11:34:40.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:34:41.001+0000 7f5f68462640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T11:34:40.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:34:41.001+0000 7f5f68462640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:34:40.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:34:41.001+0000 7f5f68462640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:34:40.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:34:41.001+0000 7f5f68462640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:34:40.998 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:34:41.001+0000 7f5f68462640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T11:34:43.844 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T11:34:43.872 DEBUG:teuthology.orchestra.run.smithi089:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:43.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:34:43.962+0000 7f5987411640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T11:34:43.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:34:43.963+0000 7f5987411640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T11:34:43.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:34:43.963+0000 7f5987411640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:34:43.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:34:43.963+0000 7f5987411640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:34:43.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:34:43.963+0000 7f5987411640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:34:43.960 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:34:43.963+0000 7f5987411640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T11:34:46.809 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T11:34:46.840 DEBUG:teuthology.orchestra.run.smithi089:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:46.931 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:34:46.934+0000 7f9d007e1640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T11:34:46.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:34:46.934+0000 7f9d007e1640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T11:34:46.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:34:46.934+0000 7f9d007e1640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:34:46.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:34:46.934+0000 7f9d007e1640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:34:46.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:34:46.934+0000 7f9d007e1640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:34:46.932 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:34:46.935+0000 7f9d007e1640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T11:34:49.786 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T11:34:49.817 DEBUG:teuthology.orchestra.run.smithi089:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:49.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:34:49.910+0000 7fee40ddd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T11:34:49.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:34:49.910+0000 7fee40ddd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T11:34:49.907 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:34:49.910+0000 7fee40ddd640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:34:49.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:34:49.910+0000 7fee40ddd640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:34:49.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:34:49.910+0000 7fee40ddd640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:34:49.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:34:49.911+0000 7fee40ddd640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T11:34:52.756 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T11:34:52.788 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T11:34:52.788 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-29T11:34:52.788 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T11:34:52.820 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-29T11:34:52.820 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T11:34:52.891 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-29T11:34:52.892 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T11:34:52.961 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-29T11:34:52.961 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T11:34:53.032 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-29T11:34:53.032 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T11:34:53.099 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-29T11:34:53.100 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T11:34:53.131 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-29T11:34:53.131 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T11:34:53.206 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-29T11:34:53.206 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T11:34:53.276 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-29T11:34:53.276 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T11:34:53.345 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-29T11:34:53.346 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T11:34:53.415 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-29T11:34:53.416 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T11:34:53.431 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-29T11:34:53.432 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T11:34:53.472 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T11:34:53.472 DEBUG:teuthology.orchestra.run.smithi072:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:34:53.474 DEBUG:teuthology.orchestra.run.smithi089:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:34:53.515 INFO:teuthology.orchestra.run.smithi072.stdout:[mgr.y] 2024-03-29T11:34:53.515 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQA+pwZmzTboHxAA4SKkpaDLa1f9H30rPT7qVw== 2024-03-29T11:34:53.515 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.0] 2024-03-29T11:34:53.515 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQBBpwZmfxtaORAAvUYWe9QVgjtv9Skc+SgMoA== 2024-03-29T11:34:53.515 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.1] 2024-03-29T11:34:53.515 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQBEpwZmNSuSNxAAOFrNVqZ45QAn8IJo50EI7w== 2024-03-29T11:34:53.515 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.2] 2024-03-29T11:34:53.515 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQBHpwZm2fJFNRAAM++Onj0Al0TuFTKLgER3TQ== 2024-03-29T11:34:53.515 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.3] 2024-03-29T11:34:53.516 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQBKpwZmhLsaNBAAmTKrQ0KK/Bq94WK+HdyzbA== 2024-03-29T11:34:53.516 INFO:teuthology.orchestra.run.smithi072.stdout:[mgr.x] 2024-03-29T11:34:53.516 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQA+pwZm7r5DJBAA5oOaZhP3F+Jhhrg+MGMAFg== 2024-03-29T11:34:53.516 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.4] 2024-03-29T11:34:53.516 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQBRpwZmqy0RABAAkOOS5pagcFl2/tvreo2Hog== 2024-03-29T11:34:53.516 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.5] 2024-03-29T11:34:53.516 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQBTpwZmqNZpORAAlJTGDUCeBgUTSGK4MJSkEw== 2024-03-29T11:34:53.516 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.6] 2024-03-29T11:34:53.516 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQBWpwZmNm26NxAAYrB002gfqBxkUCFKLgQ/lQ== 2024-03-29T11:34:53.516 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.7] 2024-03-29T11:34:53.516 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQBZpwZml+9LNhAARY8gccrqz7Hdou/qazAylQ== 2024-03-29T11:34:53.516 INFO:teuthology.orchestra.run.smithi072.stdout:[client.0] 2024-03-29T11:34:53.516 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQA+pwZmSmLxJhAAP+UIHKs/LIpaxeHzaz9DQA== 2024-03-29T11:34:53.516 INFO:teuthology.orchestra.run.smithi072.stdout:[client.1] 2024-03-29T11:34:53.516 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQA+pwZmy2tUKhAAXFiPzuJ98tQ5tP9GoWSW1w== 2024-03-29T11:34:53.538 INFO:teuthology.orchestra.run.smithi089.stdout:[mgr.y] 2024-03-29T11:34:53.538 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQA+pwZmzTboHxAA4SKkpaDLa1f9H30rPT7qVw== 2024-03-29T11:34:53.538 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.0] 2024-03-29T11:34:53.539 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQBBpwZmfxtaORAAvUYWe9QVgjtv9Skc+SgMoA== 2024-03-29T11:34:53.539 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.1] 2024-03-29T11:34:53.539 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQBEpwZmNSuSNxAAOFrNVqZ45QAn8IJo50EI7w== 2024-03-29T11:34:53.539 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.2] 2024-03-29T11:34:53.539 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQBHpwZm2fJFNRAAM++Onj0Al0TuFTKLgER3TQ== 2024-03-29T11:34:53.539 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.3] 2024-03-29T11:34:53.539 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQBKpwZmhLsaNBAAmTKrQ0KK/Bq94WK+HdyzbA== 2024-03-29T11:34:53.539 INFO:teuthology.orchestra.run.smithi089.stdout:[mgr.x] 2024-03-29T11:34:53.539 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQA+pwZm7r5DJBAA5oOaZhP3F+Jhhrg+MGMAFg== 2024-03-29T11:34:53.539 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.4] 2024-03-29T11:34:53.539 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQBRpwZmqy0RABAAkOOS5pagcFl2/tvreo2Hog== 2024-03-29T11:34:53.539 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.5] 2024-03-29T11:34:53.539 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQBTpwZmqNZpORAAlJTGDUCeBgUTSGK4MJSkEw== 2024-03-29T11:34:53.539 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.6] 2024-03-29T11:34:53.539 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQBWpwZmNm26NxAAYrB002gfqBxkUCFKLgQ/lQ== 2024-03-29T11:34:53.540 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.7] 2024-03-29T11:34:53.540 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQBZpwZml+9LNhAARY8gccrqz7Hdou/qazAylQ== 2024-03-29T11:34:53.540 INFO:teuthology.orchestra.run.smithi089.stdout:[client.0] 2024-03-29T11:34:53.540 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQA+pwZmSmLxJhAAP+UIHKs/LIpaxeHzaz9DQA== 2024-03-29T11:34:53.540 INFO:teuthology.orchestra.run.smithi089.stdout:[client.1] 2024-03-29T11:34:53.540 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQA+pwZmy2tUKhAAXFiPzuJ98tQ5tP9GoWSW1w== 2024-03-29T11:34:53.541 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:34:53.560 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:34:53.634 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:53.637 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:53.688 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:53.690 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:53.741 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:53.744 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:53.795 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:53.797 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:53.865 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:34:53.867 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:34:53.918 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:53.921 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:53.971 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:53.973 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:54.025 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:54.027 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:54.078 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:54.080 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:54.133 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:34:54.136 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:34:54.190 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:34:54.192 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:34:54.246 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T11:34:54.246 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T11:34:54.273 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:34:54.369 DEBUG:teuthology.orchestra.run.smithi072:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T11:34:54.396 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T11:34:54.463 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:34:54.560 DEBUG:teuthology.orchestra.run.smithi072:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T11:34:54.586 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T11:34:54.614 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:34:54.711 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T11:34:54.737 DEBUG:teuthology.orchestra.run.smithi072:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:54.739 DEBUG:teuthology.orchestra.run.smithi089:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:54.791 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T11:34:54.792 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T11:34:54.792 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T11:34:54.794 INFO:tasks.ceph.mon.a:Started 2024-03-29T11:34:54.794 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T11:34:54.794 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T11:34:54.796 INFO:tasks.ceph.mon.c:Started 2024-03-29T11:34:54.796 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T11:34:54.796 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T11:34:54.834 INFO:tasks.ceph.mon.b:Started 2024-03-29T11:34:54.834 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T11:34:54.834 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T11:34:54.834 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T11:34:54.836 INFO:tasks.ceph.mgr.y:Started 2024-03-29T11:34:54.836 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T11:34:54.837 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T11:34:54.838 INFO:tasks.ceph.mgr.x:Started 2024-03-29T11:34:54.839 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T11:34:54.839 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T11:34:54.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:54.942+0000 7f5711bed640 10 monclient: get_monmap_and_config 2024-03-29T11:34:54.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:54.942+0000 7f5711bed640 10 monclient: build_initial_monmap 2024-03-29T11:34:54.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:54.942+0000 7f5711bed640 10 monclient: monmap: 2024-03-29T11:34:54.940 INFO:teuthology.orchestra.run.smithi072.stderr:epoch 0 2024-03-29T11:34:54.940 INFO:teuthology.orchestra.run.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:34:54.940 INFO:teuthology.orchestra.run.smithi072.stderr:last_changed 2024-03-29T11:34:54.943560+0000 2024-03-29T11:34:54.940 INFO:teuthology.orchestra.run.smithi072.stderr:created 2024-03-29T11:34:54.943560+0000 2024-03-29T11:34:54.940 INFO:teuthology.orchestra.run.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:54.940 INFO:teuthology.orchestra.run.smithi072.stderr:election_strategy: 1 2024-03-29T11:34:54.940 INFO:teuthology.orchestra.run.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:34:54.940 INFO:teuthology.orchestra.run.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:34:54.940 INFO:teuthology.orchestra.run.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:34:54.940 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-03-29T11:34:54.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:54.943+0000 7f5711bed640 1 Processor -- start 2024-03-29T11:34:54.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:54.943+0000 7f5711bed640 1 -- start start 2024-03-29T11:34:54.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:54.943+0000 7f5711bed640 10 monclient: init 2024-03-29T11:34:54.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:54.943+0000 7f5711bed640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:54.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:54.944+0000 7f5711bed640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:34:54.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:54.944+0000 7f5711bed640 10 monclient(hunting): picked mon.noname-a con 0x7f570c0b8240 addr v1:172.21.15.72:6789/0 2024-03-29T11:34:54.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:54.944+0000 7f5711bed640 10 monclient(hunting): picked mon.noname-c con 0x7f570c0b1f60 addr v1:172.21.15.89:6789/0 2024-03-29T11:34:54.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:54.944+0000 7f5711bed640 10 monclient(hunting): picked mon.noname-b con 0x7f570c09c8c0 addr v1:172.21.15.72:6790/0 2024-03-29T11:34:54.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:54.944+0000 7f5711bed640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f570c09bd80 con 0x7f570c0b8240 2024-03-29T11:34:54.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:54.944+0000 7f5711bed640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f570c09b960 con 0x7f570c0b1f60 2024-03-29T11:34:54.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:54.944+0000 7f5711bed640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f570c0b2ec0 con 0x7f570c09c8c0 2024-03-29T11:34:54.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:54.944+0000 7f5711bed640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:54.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:54.944+0000 7f5711bed640 10 monclient(hunting): authenticate will time out at 854.232067s 2024-03-29T11:34:54.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:54.944+0000 7f570bfff640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f570c09c8c0 0x7f570c14d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:37582/0 (socket says 172.21.15.72:37582) 2024-03-29T11:34:54.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:54.944+0000 7f570b7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f570c0b8240 0x7f570c13f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:53608/0 (socket says 172.21.15.72:53608) 2024-03-29T11:34:54.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:54.944+0000 7f570bfff640 1 -- 172.21.15.72:0/1120909429 learned_addr learned my addr 172.21.15.72:0/1120909429 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:34:57.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:57.944+0000 7f57097fa640 10 monclient(hunting): tick 2024-03-29T11:34:57.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:57.944+0000 7f57097fa640 10 monclient(hunting): _check_auth_tickets 2024-03-29T11:34:57.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:57.944+0000 7f57097fa640 1 monclient(hunting): continuing hunt 2024-03-29T11:34:57.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:57.944+0000 7f57097fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T11:34:57.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:57.944+0000 7f57097fa640 1 -- 172.21.15.72:0/1120909429 >> v1:172.21.15.72:6790/0 conn(0x7f570c09c8c0 legacy=0x7f570c14d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:57.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:57.944+0000 7f57097fa640 1 -- 172.21.15.72:0/1120909429 >> v1:172.21.15.89:6789/0 conn(0x7f570c0b1f60 legacy=0x7f570c09c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:57.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:57.944+0000 7f57097fa640 1 -- 172.21.15.72:0/1120909429 >> v1:172.21.15.72:6789/0 conn(0x7f570c0b8240 legacy=0x7f570c13f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:34:57.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:57.944+0000 7f57097fa640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:34:57.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:57.944+0000 7f57097fa640 10 monclient(hunting): picked mon.noname-a con 0x7f56f80036e0 addr v1:172.21.15.72:6789/0 2024-03-29T11:34:57.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:57.944+0000 7f57097fa640 10 monclient(hunting): picked mon.noname-b con 0x7f56f80072d0 addr v1:172.21.15.72:6790/0 2024-03-29T11:34:57.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:57.944+0000 7f57097fa640 10 monclient(hunting): picked mon.noname-c con 0x7f56f800afe0 addr v1:172.21.15.89:6789/0 2024-03-29T11:34:57.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:57.944+0000 7f57097fa640 1 -- 172.21.15.72:0/1120909429 --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f570c13fb60 con 0x7f56f80036e0 2024-03-29T11:34:57.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:57.944+0000 7f57097fa640 1 -- 172.21.15.72:0/1120909429 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56f800fc90 con 0x7f56f800afe0 2024-03-29T11:34:57.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:57.944+0000 7f57097fa640 1 -- 172.21.15.72:0/1120909429 --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56f8010ef0 con 0x7f56f80072d0 2024-03-29T11:34:57.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:34:57.944+0000 7f57097fa640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:00.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.115+0000 7f570a7fc640 1 -- 172.21.15.72:0/1120909429 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2623082005 0 0) 0x7f570c13fb60 con 0x7f56f80036e0 2024-03-29T11:35:00.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.115+0000 7f570a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:00.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.115+0000 7f570a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:00.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.115+0000 7f570a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:00.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.115+0000 7f570a7fc640 10 monclient(hunting): my global_id is 4101 2024-03-29T11:35:00.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.115+0000 7f570a7fc640 1 -- 172.21.15.72:0/1120909429 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56f4003520 con 0x7f56f80036e0 2024-03-29T11:35:00.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.115+0000 7f570a7fc640 1 -- 172.21.15.72:0/1120909429 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3410341399 0 0) 0x7f56f800fc90 con 0x7f56f800afe0 2024-03-29T11:35:00.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.115+0000 7f570a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:00.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.115+0000 7f570a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:00.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.115+0000 7f570a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:00.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.115+0000 7f570a7fc640 10 monclient(hunting): my global_id is 4105 2024-03-29T11:35:00.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.115+0000 7f570a7fc640 1 -- 172.21.15.72:0/1120909429 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f570c13fb60 con 0x7f56f800afe0 2024-03-29T11:35:00.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.116+0000 7f570a7fc640 1 -- 172.21.15.72:0/1120909429 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3031522858 0 0) 0x7f570c13fb60 con 0x7f56f800afe0 2024-03-29T11:35:00.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.116+0000 7f570a7fc640 1 -- 172.21.15.72:0/1120909429 >> v1:172.21.15.72:6790/0 conn(0x7f56f80072d0 legacy=0x7f56f8009760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.116+0000 7f570a7fc640 1 -- 172.21.15.72:0/1120909429 >> v1:172.21.15.72:6789/0 conn(0x7f56f80036e0 legacy=0x7f56f8005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.116+0000 7f570a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:35:00.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.116+0000 7f570a7fc640 1 monclient: found mon.noname-c 2024-03-29T11:35:00.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.116+0000 7f570a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:00.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.116+0000 7f570a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.89:6789/0 2024-03-29T11:35:00.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.116+0000 7f570a7fc640 1 -- 172.21.15.72:0/1120909429 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f8012150 con 0x7f56f800afe0 2024-03-29T11:35:00.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.116+0000 7f570a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:35:00.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.116+0000 7f570a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:35:00.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.116+0000 7f570a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:00.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.116+0000 7f570a7fc640 1 -- 172.21.15.72:0/1120909429 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f5704001990 con 0x7f56f800afe0 2024-03-29T11:35:00.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.116+0000 7f570a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.116+0000 7f570a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:00.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.116+0000 7f570a7fc640 10 monclient: dump: 2024-03-29T11:35:00.117 INFO:teuthology.orchestra.run.smithi072.stderr:epoch 1 2024-03-29T11:35:00.117 INFO:teuthology.orchestra.run.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:00.117 INFO:teuthology.orchestra.run.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:00.117 INFO:teuthology.orchestra.run.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:00.117 INFO:teuthology.orchestra.run.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.117 INFO:teuthology.orchestra.run.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:00.117 INFO:teuthology.orchestra.run.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:00.117 INFO:teuthology.orchestra.run.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:00.117 INFO:teuthology.orchestra.run.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:00.118 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-03-29T11:35:00.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.116+0000 7f5711bed640 5 monclient: authenticate success, global_id 4105 2024-03-29T11:35:00.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.116+0000 7f5711bed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:00.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.116+0000 7f570a7fc640 1 -- 172.21.15.72:0/1120909429 <== mon.1 v1:172.21.15.89:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5704001af0 con 0x7f56f800afe0 2024-03-29T11:35:00.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.116+0000 7f570a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:00.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.116+0000 7f570a7fc640 1 -- 172.21.15.72:0/1120909429 <== mon.1 v1:172.21.15.89:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f5704001f00 con 0x7f56f800afe0 2024-03-29T11:35:00.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.116+0000 7f5711bed640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:00.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.116+0000 7f5711bed640 4 set_mon_vals no callback set 2024-03-29T11:35:00.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.116+0000 7f570a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.116+0000 7f570a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:00.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.116+0000 7f570a7fc640 10 monclient: dump: 2024-03-29T11:35:00.118 INFO:teuthology.orchestra.run.smithi072.stderr:epoch 1 2024-03-29T11:35:00.118 INFO:teuthology.orchestra.run.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:00.118 INFO:teuthology.orchestra.run.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:00.118 INFO:teuthology.orchestra.run.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:00.119 INFO:teuthology.orchestra.run.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.119 INFO:teuthology.orchestra.run.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:00.119 INFO:teuthology.orchestra.run.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:00.119 INFO:teuthology.orchestra.run.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:00.119 INFO:teuthology.orchestra.run.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:00.119 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-03-29T11:35:00.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.117+0000 7f5711bed640 10 monclient: shutdown 2024-03-29T11:35:00.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.117+0000 7f5711bed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:00.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.117+0000 7f5711bed640 1 -- 172.21.15.72:0/1120909429 >> v1:172.21.15.89:6789/0 conn(0x7f56f800afe0 legacy=0x7f56f800d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.117+0000 7f5711bed640 1 -- 172.21.15.72:0/1120909429 shutdown_connections 2024-03-29T11:35:00.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.117+0000 7f5711bed640 1 -- 172.21.15.72:0/1120909429 >> 172.21.15.72:0/1120909429 conn(0x7f570c066b00 msgr2=0x7f570c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:00.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.117+0000 7f5711bed640 1 -- 172.21.15.72:0/1120909429 shutdown_connections 2024-03-29T11:35:00.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.117+0000 7f5711bed640 1 -- 172.21.15.72:0/1120909429 wait complete. 2024-03-29T11:35:00.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.118+0000 7f5711bed640 10 monclient: build_initial_monmap 2024-03-29T11:35:00.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.118+0000 7f5711bed640 10 monclient: monmap: 2024-03-29T11:35:00.119 INFO:teuthology.orchestra.run.smithi072.stderr:epoch 0 2024-03-29T11:35:00.120 INFO:teuthology.orchestra.run.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:00.120 INFO:teuthology.orchestra.run.smithi072.stderr:last_changed 0.000000 2024-03-29T11:35:00.120 INFO:teuthology.orchestra.run.smithi072.stderr:created 0.000000 2024-03-29T11:35:00.120 INFO:teuthology.orchestra.run.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:00.120 INFO:teuthology.orchestra.run.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:00.120 INFO:teuthology.orchestra.run.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:35:00.120 INFO:teuthology.orchestra.run.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:35:00.120 INFO:teuthology.orchestra.run.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:35:00.120 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-03-29T11:35:00.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.118+0000 7f5711bed640 1 librados: starting msgr at 2024-03-29T11:35:00.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.118+0000 7f5711bed640 1 librados: starting objecter 2024-03-29T11:35:00.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.118+0000 7f5711bed640 1 Processor -- start 2024-03-29T11:35:00.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.119+0000 7f5711bed640 1 -- start start 2024-03-29T11:35:00.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.119+0000 7f5711bed640 1 librados: setting wanted keys 2024-03-29T11:35:00.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.119+0000 7f5711bed640 1 librados: calling monclient init 2024-03-29T11:35:00.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.119+0000 7f5711bed640 10 monclient: init 2024-03-29T11:35:00.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.119+0000 7f5711bed640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:00.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.119+0000 7f5711bed640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:35:00.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.119+0000 7f5711bed640 10 monclient(hunting): picked mon.noname-a con 0x7f56f80036e0 addr v1:172.21.15.72:6789/0 2024-03-29T11:35:00.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.120+0000 7f5711bed640 10 monclient(hunting): picked mon.noname-b con 0x7f56f80072d0 addr v1:172.21.15.89:6789/0 2024-03-29T11:35:00.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.120+0000 7f5711bed640 10 monclient(hunting): picked mon.noname-c con 0x7f56f800afe0 addr v1:172.21.15.72:6790/0 2024-03-29T11:35:00.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.120+0000 7f5711bed640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f570c1c9ae0 con 0x7f56f80036e0 2024-03-29T11:35:00.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.120+0000 7f570b7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f56f80036e0 0x7f570c139e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:53632/0 (socket says 172.21.15.72:53632) 2024-03-29T11:35:00.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.120+0000 7f570b7fe640 1 -- 172.21.15.72:0/2887967804 learned_addr learned my addr 172.21.15.72:0/2887967804 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:35:00.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.120+0000 7f5711bed640 1 -- 172.21.15.72:0/2887967804 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f570c1ca550 con 0x7f56f80072d0 2024-03-29T11:35:00.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.120+0000 7f5711bed640 1 -- 172.21.15.72:0/2887967804 --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f570c1ca830 con 0x7f56f800afe0 2024-03-29T11:35:00.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.120+0000 7f5711bed640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:00.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.120+0000 7f5711bed640 10 monclient(hunting): authenticate will time out at 859.408199s 2024-03-29T11:35:00.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.120+0000 7f56f3fff640 10 client.?.objecter ms_handle_connect 0x7f56f80036e0 2024-03-29T11:35:00.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.120+0000 7f56f3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:00.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.120+0000 7f56f3fff640 10 client.?.objecter ms_handle_connect 0x7f56f80072d0 2024-03-29T11:35:00.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.120+0000 7f56f3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:00.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.120+0000 7f56f3fff640 10 client.?.objecter ms_handle_connect 0x7f56f800afe0 2024-03-29T11:35:00.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.120+0000 7f56f3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:00.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.120+0000 7f570affd640 1 -- 172.21.15.72:0/2887967804 >> v1:172.21.15.89:6789/0 conn(0x7f56f80072d0 legacy=0x7f570c1da570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-29T11:35:00.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.120+0000 7f570affd640 1 -- 172.21.15.72:0/2887967804 >> v1:172.21.15.89:6789/0 conn(0x7f56f80072d0 legacy=0x7f570c1da570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:35:00.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.120+0000 7f570affd640 1 --1- 172.21.15.72:0/2887967804 >> v1:172.21.15.89:6789/0 conn(0x7f56f80072d0 0x7f570c1da570 :-1 s=OPENED pgs=19 cs=1 l=1).handle_message read tag failed 2024-03-29T11:35:00.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.120+0000 7f570affd640 1 --1- 172.21.15.72:0/2887967804 >> v1:172.21.15.89:6789/0 conn(0x7f56f80072d0 0x7f570c1da570 :-1 s=OPENED pgs=19 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:35:00.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.120+0000 7f56f3fff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.89:6789/0 2024-03-29T11:35:00.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.120+0000 7f56f3fff640 1 -- 172.21.15.72:0/2887967804 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3445013810 0 0) 0x7f570c1c9ae0 con 0x7f56f80036e0 2024-03-29T11:35:00.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.120+0000 7f56f3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:00.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.120+0000 7f56f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:00.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.120+0000 7f56f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:00.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.120+0000 7f56f3fff640 10 monclient(hunting): my global_id is 4107 2024-03-29T11:35:00.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.120+0000 7f56f3fff640 1 -- 172.21.15.72:0/2887967804 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56f8011920 con 0x7f56f80036e0 2024-03-29T11:35:00.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.121+0000 7f56f3fff640 1 -- 172.21.15.72:0/2887967804 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1160132012 0 0) 0x7f56f8011920 con 0x7f56f80036e0 2024-03-29T11:35:00.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.121+0000 7f56f3fff640 1 -- 172.21.15.72:0/2887967804 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f570c1c9ae0 con 0x7f56f80036e0 2024-03-29T11:35:00.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.121+0000 7f56f3fff640 1 -- 172.21.15.72:0/2887967804 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f56fc0018f0 con 0x7f56f80036e0 2024-03-29T11:35:00.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.121+0000 7f56f3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.121+0000 7f56f3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:35:00.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.121+0000 7f56f3fff640 10 monclient(hunting): dump: 2024-03-29T11:35:00.127 INFO:teuthology.orchestra.run.smithi072.stderr:epoch 1 2024-03-29T11:35:00.127 INFO:teuthology.orchestra.run.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:00.127 INFO:teuthology.orchestra.run.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:00.127 INFO:teuthology.orchestra.run.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:00.127 INFO:teuthology.orchestra.run.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.127 INFO:teuthology.orchestra.run.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:00.127 INFO:teuthology.orchestra.run.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:00.127 INFO:teuthology.orchestra.run.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:00.127 INFO:teuthology.orchestra.run.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:00.127 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-03-29T11:35:00.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.121+0000 7f56f3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:00.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.121+0000 7f56f3fff640 1 -- 172.21.15.72:0/2887967804 <== mon.0 v1:172.21.15.72:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 17684839 0 0) 0x7f570c1c9ae0 con 0x7f56f80036e0 2024-03-29T11:35:00.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.121+0000 7f56f3fff640 1 -- 172.21.15.72:0/2887967804 >> v1:172.21.15.72:6790/0 conn(0x7f56f800afe0 legacy=0x7f570c1dac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.121+0000 7f56f3fff640 1 -- 172.21.15.72:0/2887967804 >> v1:172.21.15.89:6789/0 conn(0x7f56f80072d0 legacy=0x7f570c1da570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:35:00.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.121+0000 7f56f3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:35:00.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.121+0000 7f56f3fff640 1 monclient: found mon.a 2024-03-29T11:35:00.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.121+0000 7f56f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:00.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.121+0000 7f56f3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:35:00.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.121+0000 7f56f3fff640 1 -- 172.21.15.72:0/2887967804 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f570c1c95c0 con 0x7f56f80036e0 2024-03-29T11:35:00.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.121+0000 7f56f3fff640 10 monclient: _finish_auth 0 2024-03-29T11:35:00.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.121+0000 7f56f3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:35:00.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.121+0000 7f56f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:00.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.121+0000 7f5711bed640 5 monclient: authenticate success, global_id 4107 2024-03-29T11:35:00.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.121+0000 7f5711bed640 10 monclient: _renew_subs 2024-03-29T11:35:00.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.121+0000 7f5711bed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:35:00.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.121+0000 7f5711bed640 1 -- 172.21.15.72:0/2887967804 --> v1:172.21.15.72:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f570c140800 con 0x7f56f80036e0 2024-03-29T11:35:00.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.121+0000 7f56f3fff640 1 -- 172.21.15.72:0/2887967804 <== mon.0 v1:172.21.15.72:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f56fc0047e0 con 0x7f56f80036e0 2024-03-29T11:35:00.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.121+0000 7f56f3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:00.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.121+0000 7f56f3fff640 1 -- 172.21.15.72:0/2887967804 <== mon.0 v1:172.21.15.72:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f56fc004af0 con 0x7f56f80036e0 2024-03-29T11:35:00.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.121+0000 7f56f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.121+0000 7f56f3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:00.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.121+0000 7f56f3fff640 10 monclient: dump: 2024-03-29T11:35:00.129 INFO:teuthology.orchestra.run.smithi072.stderr:epoch 1 2024-03-29T11:35:00.129 INFO:teuthology.orchestra.run.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:00.129 INFO:teuthology.orchestra.run.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:00.129 INFO:teuthology.orchestra.run.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:00.129 INFO:teuthology.orchestra.run.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.129 INFO:teuthology.orchestra.run.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:00.129 INFO:teuthology.orchestra.run.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:00.129 INFO:teuthology.orchestra.run.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:00.129 INFO:teuthology.orchestra.run.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:00.129 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-03-29T11:35:00.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.121+0000 7f5709ffb640 4 set_mon_vals no callback set 2024-03-29T11:35:00.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.121+0000 7f5711bed640 10 client.4107.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:00.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.121+0000 7f5711bed640 10 monclient: _renew_subs 2024-03-29T11:35:00.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.121+0000 7f5711bed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:35:00.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.121+0000 7f5711bed640 1 -- 172.21.15.72:0/2887967804 --> v1:172.21.15.72:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f570c13a710 con 0x7f56f80036e0 2024-03-29T11:35:00.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.121+0000 7f5711bed640 1 librados: init done 2024-03-29T11:35:00.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.122+0000 7f56f3fff640 1 -- 172.21.15.72:0/2887967804 <== mon.0 v1:172.21.15.72:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f56fc004fe0 con 0x7f56f80036e0 2024-03-29T11:35:00.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.122+0000 7f56f3fff640 1 -- 172.21.15.72:0/2887967804 <== mon.0 v1:172.21.15.72:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 3103891760 0 0) 0x7f56fc0064e0 con 0x7f56f80036e0 2024-03-29T11:35:00.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.122+0000 7f56f3fff640 10 client.4107.objecter ms_dispatch 0x7f570c000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T11:35:00.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.122+0000 7f56f3fff640 3 client.4107.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T11:35:00.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.122+0000 7f56f3fff640 3 client.4107.objecter handle_osd_map decoding full epoch 1 2024-03-29T11:35:00.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.122+0000 7f56f3fff640 20 client.4107.objecter dump_active .. 0 homeless 2024-03-29T11:35:00.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.123+0000 7f5711bed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:00.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.123+0000 7f5711bed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:00.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.123+0000 7f5711bed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:35:00.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.123+0000 7f5711bed640 1 -- 172.21.15.72:0/2887967804 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f570c09b170 con 0x7f56f80036e0 2024-03-29T11:35:00.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.128+0000 7f56f3fff640 1 -- 172.21.15.72:0/2887967804 <== mon.0 v1:172.21.15.72:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f56fc005430 con 0x7f56f80036e0 2024-03-29T11:35:00.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.128+0000 7f56f3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:00.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.128+0000 7f56f3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:00.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.198+0000 7f5711bed640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:35:00.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.198+0000 7f5711bed640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:35:00.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.198+0000 7f5711bed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:35:00.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.198+0000 7f5711bed640 1 -- 172.21.15.72:0/2887967804 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f570c1cad50 con 0x7f56f80036e0 2024-03-29T11:35:00.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.199+0000 7f56f3fff640 1 -- 172.21.15.72:0/2887967804 <== mon.0 v1:172.21.15.72:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f570c1cad50 con 0x7f56f80036e0 2024-03-29T11:35:00.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.199+0000 7f56f3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:35:00.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.199+0000 7f56f3fff640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T11:35:00.197 INFO:teuthology.orchestra.run.smithi072.stderr:adjusted tunables profile to default 2024-03-29T11:35:00.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.200+0000 7f5711bed640 10 librados: watch_flush enter 2024-03-29T11:35:00.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.200+0000 7f5711bed640 10 librados: watch_flush exit 2024-03-29T11:35:00.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.200+0000 7f5711bed640 20 client.4107.objecter shutdown clearing up homeless session... 2024-03-29T11:35:00.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.200+0000 7f5711bed640 10 client.4107.objecter successfully canceled tick 2024-03-29T11:35:00.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.200+0000 7f5711bed640 10 monclient: shutdown 2024-03-29T11:35:00.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.200+0000 7f5711bed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:00.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.200+0000 7f5711bed640 1 -- 172.21.15.72:0/2887967804 >> v1:172.21.15.72:6789/0 conn(0x7f56f80036e0 legacy=0x7f570c139e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.200+0000 7f5711bed640 1 -- 172.21.15.72:0/2887967804 shutdown_connections 2024-03-29T11:35:00.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.200+0000 7f5711bed640 1 -- 172.21.15.72:0/2887967804 >> 172.21.15.72:0/2887967804 conn(0x7f570c066b00 msgr2=0x7f570c0b5890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:00.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.200+0000 7f5711bed640 1 -- 172.21.15.72:0/2887967804 shutdown_connections 2024-03-29T11:35:00.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.200+0000 7f5711bed640 1 -- 172.21.15.72:0/2887967804 wait complete. 2024-03-29T11:35:00.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:00.200+0000 7f5711bed640 1 librados: shutdown 2024-03-29T11:35:00.209 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T11:35:00.209 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T11:35:00.209 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-29T11:35:00.210 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T11:35:00.237 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-29T11:35:00.237 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T11:35:00.303 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-29T11:35:00.303 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T11:35:00.374 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-29T11:35:00.374 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T11:35:00.441 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-29T11:35:00.441 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T11:35:00.469 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-29T11:35:00.469 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T11:35:00.536 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-29T11:35:00.536 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T11:35:00.593 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-29T11:35:00.595+0000 7f1f7fc2b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:35:00.602 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-03-29T11:35:00.603 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T11:35:00.639 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-29T11:35:00.642+0000 7fc853e3e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:35:00.649 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-29T11:35:00.651+0000 7f1f7fc2b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:35:00.670 DEBUG:teuthology.orchestra.run.smithi089:> sudo ceph --cluster ceph osd new a2a7cb44-331d-432e-a832-f28309f32297 0 2024-03-29T11:35:00.693 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-29T11:35:00.695+0000 7fc853e3e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:35:00.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.812+0000 7f7d1358d640 10 monclient: get_monmap_and_config 2024-03-29T11:35:00.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.812+0000 7f7d1358d640 10 monclient: build_initial_monmap 2024-03-29T11:35:00.810 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.812+0000 7f7d1358d640 10 monclient: monmap: 2024-03-29T11:35:00.811 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 0 2024-03-29T11:35:00.811 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:00.811 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:35:00.813484+0000 2024-03-29T11:35:00.811 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:35:00.813484+0000 2024-03-29T11:35:00.811 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:00.811 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:00.811 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:35:00.811 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:35:00.811 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:35:00.811 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:00.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.812+0000 7f7d1358d640 1 Processor -- start 2024-03-29T11:35:00.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.812+0000 7f7d1358d640 1 -- start start 2024-03-29T11:35:00.811 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.812+0000 7f7d1358d640 10 monclient: init 2024-03-29T11:35:00.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.813+0000 7f7d1358d640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:00.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.813+0000 7f7d1358d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:35:00.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.813+0000 7f7d1358d640 10 monclient(hunting): picked mon.noname-a con 0x7f7d0c09ac40 addr v1:172.21.15.72:6789/0 2024-03-29T11:35:00.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.813+0000 7f7d1358d640 10 monclient(hunting): picked mon.noname-c con 0x7f7d0c09c870 addr v1:172.21.15.89:6789/0 2024-03-29T11:35:00.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.813+0000 7f7d1358d640 10 monclient(hunting): picked mon.noname-b con 0x7f7d0c0b4360 addr v1:172.21.15.72:6790/0 2024-03-29T11:35:00.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.813+0000 7f7d1358d640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d0c09dec0 con 0x7f7d0c09ac40 2024-03-29T11:35:00.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.813+0000 7f7d1358d640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d0c09da30 con 0x7f7d0c09c870 2024-03-29T11:35:00.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.813+0000 7f7d1358d640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d0c0b39c0 con 0x7f7d0c0b4360 2024-03-29T11:35:00.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.813+0000 7f7d1358d640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:00.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.813+0000 7f7d1358d640 10 monclient(hunting): authenticate will time out at 873.086463s 2024-03-29T11:35:00.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.813+0000 7f7d10b28640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f7d0c09c870 0x7f7d0c13f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:56986/0 (socket says 172.21.15.89:56986) 2024-03-29T11:35:00.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.813+0000 7f7d10b28640 1 -- 172.21.15.89:0/119749193 learned_addr learned my addr 172.21.15.89:0/119749193 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-03-29T11:35:00.821 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.813+0000 7f7d03fff640 1 -- 172.21.15.89:0/119749193 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 722790423 0 0) 0x7f7d0c09da30 con 0x7f7d0c09c870 2024-03-29T11:35:00.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.813+0000 7f7d03fff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:00.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.813+0000 7f7d03fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:00.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.813+0000 7f7d03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:00.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.813+0000 7f7d03fff640 10 monclient(hunting): my global_id is 4114 2024-03-29T11:35:00.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.813+0000 7f7d03fff640 1 -- 172.21.15.89:0/119749193 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cf8003520 con 0x7f7d0c09c870 2024-03-29T11:35:00.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.813+0000 7f7d03fff640 1 -- 172.21.15.89:0/119749193 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1000653600 0 0) 0x7f7d0c0b39c0 con 0x7f7d0c0b4360 2024-03-29T11:35:00.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.813+0000 7f7d03fff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:00.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.813+0000 7f7d03fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:00.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.813+0000 7f7d03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:00.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.813+0000 7f7d03fff640 10 monclient(hunting): my global_id is 4103 2024-03-29T11:35:00.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.813+0000 7f7d03fff640 1 -- 172.21.15.89:0/119749193 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d0c09da30 con 0x7f7d0c0b4360 2024-03-29T11:35:00.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.813+0000 7f7d03fff640 1 -- 172.21.15.89:0/119749193 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2308291310 0 0) 0x7f7cfc001110 con 0x7f7d0c09ac40 2024-03-29T11:35:00.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.813+0000 7f7d03fff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:00.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.813+0000 7f7d03fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:00.822 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.813+0000 7f7d03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:00.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.813+0000 7f7d03fff640 10 monclient(hunting): my global_id is 4110 2024-03-29T11:35:00.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.813+0000 7f7d03fff640 1 -- 172.21.15.89:0/119749193 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d0c0b39c0 con 0x7f7d0c09ac40 2024-03-29T11:35:00.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.814+0000 7f7d03fff640 1 -- 172.21.15.89:0/119749193 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 470631746 0 0) 0x7f7cf8003520 con 0x7f7d0c09c870 2024-03-29T11:35:00.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.814+0000 7f7d03fff640 1 -- 172.21.15.89:0/119749193 >> v1:172.21.15.72:6790/0 conn(0x7f7d0c0b4360 legacy=0x7f7d0c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.814+0000 7f7d03fff640 1 -- 172.21.15.89:0/119749193 >> v1:172.21.15.72:6789/0 conn(0x7f7d0c09ac40 legacy=0x7f7d0c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.814+0000 7f7d03fff640 10 monclient: _finish_hunting 0 2024-03-29T11:35:00.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.814+0000 7f7d03fff640 1 monclient: found mon.noname-c 2024-03-29T11:35:00.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.814+0000 7f7d03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:00.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.814+0000 7f7d03fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.89:6789/0 2024-03-29T11:35:00.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.814+0000 7f7d03fff640 1 -- 172.21.15.89:0/119749193 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d0c0b5160 con 0x7f7d0c09c870 2024-03-29T11:35:00.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.814+0000 7f7d03fff640 10 monclient: _finish_auth 0 2024-03-29T11:35:00.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.814+0000 7f7d03fff640 10 monclient: _check_auth_tickets 2024-03-29T11:35:00.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.814+0000 7f7d03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:00.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.814+0000 7f7d03fff640 1 -- 172.21.15.89:0/119749193 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f7d08001690 con 0x7f7d0c09c870 2024-03-29T11:35:00.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.814+0000 7f7d03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.823 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.814+0000 7f7d03fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:00.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.814+0000 7f7d03fff640 10 monclient: dump: 2024-03-29T11:35:00.824 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:00.824 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:00.824 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:00.824 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:00.824 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.824 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:00.824 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:00.824 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:00.824 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:00.824 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:00.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.814+0000 7f7d1358d640 5 monclient: authenticate success, global_id 4114 2024-03-29T11:35:00.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.814+0000 7f7d1358d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:00.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.814+0000 7f7d03fff640 1 -- 172.21.15.89:0/119749193 <== mon.1 v1:172.21.15.89:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d08001870 con 0x7f7d0c09c870 2024-03-29T11:35:00.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.814+0000 7f7d03fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:00.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.814+0000 7f7d03fff640 1 -- 172.21.15.89:0/119749193 <== mon.1 v1:172.21.15.89:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f7d08001c80 con 0x7f7d0c09c870 2024-03-29T11:35:00.824 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.814+0000 7f7d1358d640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:00.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.814+0000 7f7d1358d640 4 set_mon_vals no callback set 2024-03-29T11:35:00.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.814+0000 7f7d03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.814+0000 7f7d03fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:00.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.814+0000 7f7d03fff640 10 monclient: dump: 2024-03-29T11:35:00.825 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:00.825 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:00.825 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:00.825 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:00.825 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.825 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:00.825 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:00.825 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:00.825 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:00.825 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:00.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.814+0000 7f7d1358d640 10 monclient: shutdown 2024-03-29T11:35:00.825 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.814+0000 7f7d1358d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:00.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.814+0000 7f7d1358d640 1 -- 172.21.15.89:0/119749193 >> v1:172.21.15.89:6789/0 conn(0x7f7d0c09c870 legacy=0x7f7d0c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.814+0000 7f7d1358d640 1 -- 172.21.15.89:0/119749193 shutdown_connections 2024-03-29T11:35:00.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.814+0000 7f7d1358d640 1 -- 172.21.15.89:0/119749193 >> 172.21.15.89:0/119749193 conn(0x7f7d0c066b00 msgr2=0x7f7d0c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:00.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.814+0000 7f7d1358d640 1 -- 172.21.15.89:0/119749193 shutdown_connections 2024-03-29T11:35:00.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.814+0000 7f7d1358d640 1 -- 172.21.15.89:0/119749193 wait complete. 2024-03-29T11:35:00.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.815+0000 7f7d1358d640 10 monclient: build_initial_monmap 2024-03-29T11:35:00.826 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.815+0000 7f7d1358d640 10 monclient: monmap: 2024-03-29T11:35:00.826 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 0 2024-03-29T11:35:00.826 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:00.826 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 0.000000 2024-03-29T11:35:00.826 INFO:teuthology.orchestra.run.smithi089.stderr:created 0.000000 2024-03-29T11:35:00.826 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:00.826 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:00.826 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:35:00.826 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:35:00.827 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:35:00.827 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:00.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.815+0000 7f7d1358d640 1 librados: starting msgr at 2024-03-29T11:35:00.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.815+0000 7f7d1358d640 1 librados: starting objecter 2024-03-29T11:35:00.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.815+0000 7f7d1358d640 1 Processor -- start 2024-03-29T11:35:00.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.815+0000 7f7d1358d640 1 -- start start 2024-03-29T11:35:00.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.815+0000 7f7d1358d640 1 librados: setting wanted keys 2024-03-29T11:35:00.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.815+0000 7f7d1358d640 1 librados: calling monclient init 2024-03-29T11:35:00.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.815+0000 7f7d1358d640 10 monclient: init 2024-03-29T11:35:00.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.816+0000 7f7d1358d640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:00.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.816+0000 7f7d1358d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:35:00.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.816+0000 7f7d1358d640 10 monclient(hunting): picked mon.noname-b con 0x7f7d0c09ac40 addr v1:172.21.15.89:6789/0 2024-03-29T11:35:00.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.816+0000 7f7d1358d640 10 monclient(hunting): picked mon.noname-a con 0x7f7d0c0b07a0 addr v1:172.21.15.72:6789/0 2024-03-29T11:35:00.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.816+0000 7f7d1358d640 10 monclient(hunting): picked mon.noname-c con 0x7f7d0c1d7cd0 addr v1:172.21.15.72:6790/0 2024-03-29T11:35:00.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.816+0000 7f7d1358d640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d0c0acd70 con 0x7f7d0c0b07a0 2024-03-29T11:35:00.827 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.816+0000 7f7d1358d640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d0c1d8100 con 0x7f7d0c09ac40 2024-03-29T11:35:00.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.816+0000 7f7d1358d640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d0c11f8d0 con 0x7f7d0c1d7cd0 2024-03-29T11:35:00.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.816+0000 7f7d1358d640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:00.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.816+0000 7f7d1358d640 10 monclient(hunting): authenticate will time out at 873.089490s 2024-03-29T11:35:00.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.816+0000 7f7d11329640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f7d0c09ac40 0x7f7d0c0afa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57000/0 (socket says 172.21.15.89:57000) 2024-03-29T11:35:00.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.816+0000 7f7d11329640 1 -- 172.21.15.89:0/55251079 learned_addr learned my addr 172.21.15.89:0/55251079 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-03-29T11:35:00.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.816+0000 7f7d01ffb640 10 client.?.objecter ms_handle_connect 0x7f7d0c09ac40 2024-03-29T11:35:00.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.816+0000 7f7d01ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:00.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.816+0000 7f7d01ffb640 10 client.?.objecter ms_handle_connect 0x7f7d0c0b07a0 2024-03-29T11:35:00.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.816+0000 7f7d01ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:00.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.816+0000 7f7d01ffb640 10 client.?.objecter ms_handle_connect 0x7f7d0c1d7cd0 2024-03-29T11:35:00.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.816+0000 7f7d01ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:00.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.816+0000 7f7d01ffb640 1 -- 172.21.15.89:0/55251079 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2081865909 0 0) 0x7f7d0c1d8100 con 0x7f7d0c09ac40 2024-03-29T11:35:00.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.816+0000 7f7d01ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:00.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.816+0000 7f7d01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:00.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.816+0000 7f7d01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:00.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.816+0000 7f7d01ffb640 10 monclient(hunting): my global_id is 4117 2024-03-29T11:35:00.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.816+0000 7f7d01ffb640 1 -- 172.21.15.89:0/55251079 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cf0004eb0 con 0x7f7d0c09ac40 2024-03-29T11:35:00.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.816+0000 7f7d01ffb640 1 -- 172.21.15.89:0/55251079 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3607709239 0 0) 0x7f7d0c11f8d0 con 0x7f7d0c1d7cd0 2024-03-29T11:35:00.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.816+0000 7f7d01ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:00.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.816+0000 7f7d01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:00.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.816+0000 7f7d01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:00.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.816+0000 7f7d01ffb640 10 monclient(hunting): my global_id is 4106 2024-03-29T11:35:00.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.816+0000 7f7d01ffb640 1 -- 172.21.15.89:0/55251079 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d0c1d8100 con 0x7f7d0c1d7cd0 2024-03-29T11:35:00.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.816+0000 7f7d01ffb640 1 -- 172.21.15.89:0/55251079 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3527652967 0 0) 0x7f7d0c0acd70 con 0x7f7d0c0b07a0 2024-03-29T11:35:00.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.816+0000 7f7d01ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:00.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.816+0000 7f7d01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:00.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.816+0000 7f7d01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:00.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.816+0000 7f7d01ffb640 10 monclient(hunting): my global_id is 4113 2024-03-29T11:35:00.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.816+0000 7f7d01ffb640 1 -- 172.21.15.89:0/55251079 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d0c11f8d0 con 0x7f7d0c0b07a0 2024-03-29T11:35:00.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d01ffb640 1 -- 172.21.15.89:0/55251079 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2687053634 0 0) 0x7f7cf0004eb0 con 0x7f7d0c09ac40 2024-03-29T11:35:00.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d01ffb640 1 -- 172.21.15.89:0/55251079 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d0c0acd70 con 0x7f7d0c09ac40 2024-03-29T11:35:00.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d01ffb640 1 -- 172.21.15.89:0/55251079 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f7cfc001860 con 0x7f7d0c09ac40 2024-03-29T11:35:00.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d01ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d01ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:00.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d01ffb640 10 monclient(hunting): dump: 2024-03-29T11:35:00.830 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:00.830 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:00.830 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:00.830 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:00.830 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.830 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:00.830 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:00.830 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:00.830 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:00.830 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:00.830 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d01ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:00.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d01ffb640 1 -- 172.21.15.89:0/55251079 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1225257763 0 0) 0x7f7d0c1d8100 con 0x7f7d0c1d7cd0 2024-03-29T11:35:00.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d01ffb640 1 -- 172.21.15.89:0/55251079 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7cf0004eb0 con 0x7f7d0c1d7cd0 2024-03-29T11:35:00.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d01ffb640 1 -- 172.21.15.89:0/55251079 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2884026692 0 0) 0x7f7d0c11f8d0 con 0x7f7d0c0b07a0 2024-03-29T11:35:00.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d01ffb640 1 -- 172.21.15.89:0/55251079 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d0c1d8100 con 0x7f7d0c0b07a0 2024-03-29T11:35:00.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d01ffb640 1 -- 172.21.15.89:0/55251079 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f7d04001a10 con 0x7f7d0c1d7cd0 2024-03-29T11:35:00.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d01ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d01ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:00.831 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d01ffb640 10 monclient(hunting): dump: 2024-03-29T11:35:00.831 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:00.831 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:00.831 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:00.831 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:00.831 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.831 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:00.831 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:00.832 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:00.832 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:00.832 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:00.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d01ffb640 1 -- 172.21.15.89:0/55251079 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f7d08001b80 con 0x7f7d0c0b07a0 2024-03-29T11:35:00.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d01ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d01ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:00.832 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d01ffb640 10 monclient(hunting): dump: 2024-03-29T11:35:00.832 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:00.832 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:00.832 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:00.832 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:00.832 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.832 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:00.832 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:00.832 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:00.832 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:00.833 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:00.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d01ffb640 1 -- 172.21.15.89:0/55251079 <== mon.1 v1:172.21.15.89:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1294499635 0 0) 0x7f7d0c0acd70 con 0x7f7d0c09ac40 2024-03-29T11:35:00.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d01ffb640 1 -- 172.21.15.89:0/55251079 >> v1:172.21.15.72:6790/0 conn(0x7f7d0c1d7cd0 legacy=0x7f7d0c1e1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d01ffb640 1 -- 172.21.15.89:0/55251079 >> v1:172.21.15.72:6789/0 conn(0x7f7d0c0b07a0 legacy=0x7f7d0c1d75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d01ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:35:00.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d01ffb640 1 monclient: found mon.b 2024-03-29T11:35:00.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d01ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:00.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d01ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:00.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d01ffb640 1 -- 172.21.15.89:0/55251079 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d0c1e64c0 con 0x7f7d0c09ac40 2024-03-29T11:35:00.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d01ffb640 10 monclient: _finish_auth 0 2024-03-29T11:35:00.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d01ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:35:00.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d01ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:00.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d1358d640 5 monclient: authenticate success, global_id 4117 2024-03-29T11:35:00.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d1358d640 10 monclient: _renew_subs 2024-03-29T11:35:00.833 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d1358d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:00.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d1358d640 1 -- 172.21.15.89:0/55251079 --> v1:172.21.15.89:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d0c1e3490 con 0x7f7d0c09ac40 2024-03-29T11:35:00.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d01ffb640 1 -- 172.21.15.89:0/55251079 <== mon.1 v1:172.21.15.89:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7cfc003610 con 0x7f7d0c09ac40 2024-03-29T11:35:00.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d01ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:00.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d01ffb640 1 -- 172.21.15.89:0/55251079 <== mon.1 v1:172.21.15.89:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f7cfc003a20 con 0x7f7d0c09ac40 2024-03-29T11:35:00.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:00.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d01ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:00.834 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d01ffb640 10 monclient: dump: 2024-03-29T11:35:00.834 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:00.834 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:00.834 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:00.834 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:00.834 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:00.834 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:00.834 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:00.834 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:00.835 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:00.835 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:00.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d1358d640 10 client.4117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:00.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d037fe640 4 set_mon_vals no callback set 2024-03-29T11:35:00.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d1358d640 10 monclient: _renew_subs 2024-03-29T11:35:00.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d1358d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:00.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d1358d640 1 -- 172.21.15.89:0/55251079 --> v1:172.21.15.89:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d0c1e3970 con 0x7f7d0c09ac40 2024-03-29T11:35:00.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d01ffb640 1 -- 172.21.15.89:0/55251079 <== mon.1 v1:172.21.15.89:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f7cfc003f90 con 0x7f7d0c09ac40 2024-03-29T11:35:00.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d1358d640 1 librados: init done 2024-03-29T11:35:00.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d01ffb640 1 -- 172.21.15.89:0/55251079 <== mon.1 v1:172.21.15.89:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 3634751142 0 0) 0x7f7cfc005b40 con 0x7f7d0c09ac40 2024-03-29T11:35:00.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d01ffb640 10 client.4117.objecter ms_dispatch 0x7f7d0c000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T11:35:00.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d01ffb640 3 client.4117.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T11:35:00.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d01ffb640 3 client.4117.objecter handle_osd_map decoding full epoch 2 2024-03-29T11:35:00.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.817+0000 7f7d01ffb640 20 client.4117.objecter dump_active .. 0 homeless 2024-03-29T11:35:00.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.818+0000 7f7d1358d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:00.835 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.818+0000 7f7d1358d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:00.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.818+0000 7f7d1358d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:00.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.818+0000 7f7d1358d640 1 -- 172.21.15.89:0/55251079 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d0c09b020 con 0x7f7d0c09ac40 2024-03-29T11:35:00.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.820+0000 7f7d01ffb640 1 -- 172.21.15.89:0/55251079 <== mon.1 v1:172.21.15.89:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f7cfc0057c0 con 0x7f7d0c09ac40 2024-03-29T11:35:00.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.820+0000 7f7d01ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:00.836 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.820+0000 7f7d01ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:00.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.891+0000 7f7d1358d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a2a7cb44-331d-432e-a832-f28309f32297", "id": 0}] 2024-03-29T11:35:00.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.891+0000 7f7d1358d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a2a7cb44-331d-432e-a832-f28309f32297", "id": 0}] 2024-03-29T11:35:00.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.892+0000 7f7d1358d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:00.890 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.892+0000 7f7d1358d640 1 -- 172.21.15.89:0/55251079 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "osd new", "uuid": "a2a7cb44-331d-432e-a832-f28309f32297", "id": 0} v 0) v1 -- 0x7f7d0c04f730 con 0x7f7d0c09ac40 2024-03-29T11:35:00.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.897+0000 7f7d01ffb640 1 -- 172.21.15.89:0/55251079 <== mon.1 v1:172.21.15.89:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a2a7cb44-331d-432e-a832-f28309f32297", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 3579760603 0 732707515) 0x7f7cfc006360 con 0x7f7d0c09ac40 2024-03-29T11:35:00.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.897+0000 7f7d01ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a2a7cb44-331d-432e-a832-f28309f32297", "id": 0}] 2024-03-29T11:35:00.895 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.897+0000 7f7d01ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:00.895 INFO:teuthology.orchestra.run.smithi089.stdout:0 2024-03-29T11:35:00.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.898+0000 7f7d1358d640 10 librados: watch_flush enter 2024-03-29T11:35:00.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.898+0000 7f7d1358d640 10 librados: watch_flush exit 2024-03-29T11:35:00.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.898+0000 7f7d1358d640 20 client.4117.objecter shutdown clearing up homeless session... 2024-03-29T11:35:00.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.898+0000 7f7d1358d640 10 client.4117.objecter successfully canceled tick 2024-03-29T11:35:00.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.898+0000 7f7d1358d640 10 monclient: shutdown 2024-03-29T11:35:00.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.898+0000 7f7d1358d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:00.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.898+0000 7f7d1358d640 1 -- 172.21.15.89:0/55251079 >> v1:172.21.15.89:6789/0 conn(0x7f7d0c09ac40 legacy=0x7f7d0c0afa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:00.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.898+0000 7f7d1358d640 1 -- 172.21.15.89:0/55251079 shutdown_connections 2024-03-29T11:35:00.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.898+0000 7f7d1358d640 1 -- 172.21.15.89:0/55251079 >> 172.21.15.89:0/55251079 conn(0x7f7d0c066b00 msgr2=0x7f7d0c0b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:00.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.898+0000 7f7d1358d640 1 -- 172.21.15.89:0/55251079 shutdown_connections 2024-03-29T11:35:00.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.898+0000 7f7d1358d640 1 -- 172.21.15.89:0/55251079 wait complete. 2024-03-29T11:35:00.897 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:00.898+0000 7f7d1358d640 1 librados: shutdown 2024-03-29T11:35:00.906 DEBUG:teuthology.orchestra.run.smithi089:> sudo ceph --cluster ceph osd new e054ff7c-3661-44b2-b369-5dbf77516540 1 2024-03-29T11:35:01.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.007+0000 7f0e7ba33640 10 monclient: get_monmap_and_config 2024-03-29T11:35:01.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.007+0000 7f0e7ba33640 10 monclient: build_initial_monmap 2024-03-29T11:35:01.005 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.007+0000 7f0e7ba33640 10 monclient: monmap: 2024-03-29T11:35:01.005 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 0 2024-03-29T11:35:01.005 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:01.005 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:35:01.008454+0000 2024-03-29T11:35:01.005 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:35:01.008454+0000 2024-03-29T11:35:01.005 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:01.005 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:01.006 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:35:01.006 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:35:01.006 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:35:01.006 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:01.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.007+0000 7f0e7ba33640 1 Processor -- start 2024-03-29T11:35:01.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.007+0000 7f0e7ba33640 1 -- start start 2024-03-29T11:35:01.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.007+0000 7f0e7ba33640 10 monclient: init 2024-03-29T11:35:01.015 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.008+0000 7f0e7ba33640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:01.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.008+0000 7f0e7ba33640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:35:01.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.008+0000 7f0e7ba33640 10 monclient(hunting): picked mon.noname-b con 0x7f0e74066b90 addr v1:172.21.15.72:6790/0 2024-03-29T11:35:01.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.008+0000 7f0e7ba33640 10 monclient(hunting): picked mon.noname-c con 0x7f0e7413f5e0 addr v1:172.21.15.89:6789/0 2024-03-29T11:35:01.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.008+0000 7f0e7ba33640 10 monclient(hunting): picked mon.noname-a con 0x7f0e741426a0 addr v1:172.21.15.72:6789/0 2024-03-29T11:35:01.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.008+0000 7f0e7ba33640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e740b1090 con 0x7f0e741426a0 2024-03-29T11:35:01.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.008+0000 7f0e7ba33640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e74142b30 con 0x7f0e7413f5e0 2024-03-29T11:35:01.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.008+0000 7f0e7ba33640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e740b1300 con 0x7f0e74066b90 2024-03-29T11:35:01.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.008+0000 7f0e7ba33640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:01.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.008+0000 7f0e7ba33640 10 monclient(hunting): authenticate will time out at 873.281447s 2024-03-29T11:35:01.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.008+0000 7f0e78fce640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f0e7413f5e0 0x7f0e74141f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57008/0 (socket says 172.21.15.89:57008) 2024-03-29T11:35:01.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.008+0000 7f0e78fce640 1 -- 172.21.15.89:0/3056203916 learned_addr learned my addr 172.21.15.89:0/3056203916 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-03-29T11:35:01.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.008+0000 7f0e6bfff640 1 -- 172.21.15.89:0/3056203916 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1279115524 0 0) 0x7f0e74142b30 con 0x7f0e7413f5e0 2024-03-29T11:35:01.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.008+0000 7f0e6bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.008+0000 7f0e6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.016 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.008+0000 7f0e6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.008+0000 7f0e6bfff640 10 monclient(hunting): my global_id is 4120 2024-03-29T11:35:01.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.008+0000 7f0e6bfff640 1 -- 172.21.15.89:0/3056203916 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e60003520 con 0x7f0e7413f5e0 2024-03-29T11:35:01.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.008+0000 7f0e6bfff640 1 -- 172.21.15.89:0/3056203916 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1968069563 0 0) 0x7f0e740b1300 con 0x7f0e74066b90 2024-03-29T11:35:01.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.008+0000 7f0e6bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.008+0000 7f0e6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.008+0000 7f0e6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.008+0000 7f0e6bfff640 10 monclient(hunting): my global_id is 4109 2024-03-29T11:35:01.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.008+0000 7f0e6bfff640 1 -- 172.21.15.89:0/3056203916 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e74142b30 con 0x7f0e74066b90 2024-03-29T11:35:01.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.008+0000 7f0e6bfff640 1 -- 172.21.15.89:0/3056203916 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3655715417 0 0) 0x7f0e740b1090 con 0x7f0e741426a0 2024-03-29T11:35:01.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.008+0000 7f0e6bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.008+0000 7f0e6bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.008+0000 7f0e6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.008+0000 7f0e6bfff640 10 monclient(hunting): my global_id is 4116 2024-03-29T11:35:01.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.008+0000 7f0e6bfff640 1 -- 172.21.15.89:0/3056203916 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e740b1300 con 0x7f0e741426a0 2024-03-29T11:35:01.017 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.009+0000 7f0e6bfff640 1 -- 172.21.15.89:0/3056203916 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3867941815 0 0) 0x7f0e60003520 con 0x7f0e7413f5e0 2024-03-29T11:35:01.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.009+0000 7f0e6bfff640 1 -- 172.21.15.89:0/3056203916 >> v1:172.21.15.72:6790/0 conn(0x7f0e74066b90 legacy=0x7f0e7413eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.009+0000 7f0e6bfff640 1 -- 172.21.15.89:0/3056203916 >> v1:172.21.15.72:6789/0 conn(0x7f0e741426a0 legacy=0x7f0e7414d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.009+0000 7f0e6bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:35:01.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.009+0000 7f0e6bfff640 1 monclient: found mon.noname-c 2024-03-29T11:35:01.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.009+0000 7f0e6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:01.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.009+0000 7f0e6bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.89:6789/0 2024-03-29T11:35:01.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.009+0000 7f0e6bfff640 1 -- 172.21.15.89:0/3056203916 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e74067300 con 0x7f0e7413f5e0 2024-03-29T11:35:01.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.009+0000 7f0e6bfff640 10 monclient: _finish_auth 0 2024-03-29T11:35:01.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.009+0000 7f0e6bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:35:01.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.009+0000 7f0e6bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:01.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.009+0000 7f0e7ba33640 5 monclient: authenticate success, global_id 4120 2024-03-29T11:35:01.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.009+0000 7f0e7ba33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:01.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.009+0000 7f0e6bfff640 1 -- 172.21.15.89:0/3056203916 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f0e70001690 con 0x7f0e7413f5e0 2024-03-29T11:35:01.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.009+0000 7f0e6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.018 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.009+0000 7f0e6bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:01.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.009+0000 7f0e6bfff640 10 monclient: dump: 2024-03-29T11:35:01.019 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:01.019 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:01.019 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.019 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.019 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.019 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:01.019 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:01.019 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:01.019 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:01.019 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:01.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.009+0000 7f0e7ba33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:01.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.009+0000 7f0e6bfff640 1 -- 172.21.15.89:0/3056203916 <== mon.1 v1:172.21.15.89:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e700017f0 con 0x7f0e7413f5e0 2024-03-29T11:35:01.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.009+0000 7f0e6bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:01.019 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.009+0000 7f0e6bfff640 1 -- 172.21.15.89:0/3056203916 <== mon.1 v1:172.21.15.89:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f0e70001bc0 con 0x7f0e7413f5e0 2024-03-29T11:35:01.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.009+0000 7f0e6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.009+0000 7f0e6bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:01.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.009+0000 7f0e6bfff640 10 monclient: dump: 2024-03-29T11:35:01.020 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:01.020 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:01.020 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.020 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.020 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.020 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:01.020 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:01.020 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:01.020 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:01.020 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:01.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.009+0000 7f0e7ba33640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:01.020 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.009+0000 7f0e7ba33640 4 set_mon_vals no callback set 2024-03-29T11:35:01.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.009+0000 7f0e7ba33640 10 monclient: shutdown 2024-03-29T11:35:01.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.009+0000 7f0e7ba33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:01.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.009+0000 7f0e7ba33640 1 -- 172.21.15.89:0/3056203916 >> v1:172.21.15.89:6789/0 conn(0x7f0e7413f5e0 legacy=0x7f0e74141f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.009+0000 7f0e7ba33640 1 -- 172.21.15.89:0/3056203916 shutdown_connections 2024-03-29T11:35:01.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.009+0000 7f0e7ba33640 1 -- 172.21.15.89:0/3056203916 >> 172.21.15.89:0/3056203916 conn(0x7f0e740b4a50 msgr2=0x7f0e740b4e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:01.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.010+0000 7f0e7ba33640 1 -- 172.21.15.89:0/3056203916 shutdown_connections 2024-03-29T11:35:01.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.010+0000 7f0e7ba33640 1 -- 172.21.15.89:0/3056203916 wait complete. 2024-03-29T11:35:01.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.010+0000 7f0e7ba33640 10 monclient: build_initial_monmap 2024-03-29T11:35:01.021 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.010+0000 7f0e7ba33640 10 monclient: monmap: 2024-03-29T11:35:01.021 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 0 2024-03-29T11:35:01.021 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:01.021 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 0.000000 2024-03-29T11:35:01.021 INFO:teuthology.orchestra.run.smithi089.stderr:created 0.000000 2024-03-29T11:35:01.021 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:01.021 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:01.022 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:35:01.022 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:35:01.022 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:35:01.022 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:01.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.010+0000 7f0e7ba33640 1 librados: starting msgr at 2024-03-29T11:35:01.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.010+0000 7f0e7ba33640 1 librados: starting objecter 2024-03-29T11:35:01.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.010+0000 7f0e7ba33640 1 Processor -- start 2024-03-29T11:35:01.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.011+0000 7f0e7ba33640 1 -- start start 2024-03-29T11:35:01.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.011+0000 7f0e7ba33640 1 librados: setting wanted keys 2024-03-29T11:35:01.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.011+0000 7f0e7ba33640 1 librados: calling monclient init 2024-03-29T11:35:01.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.011+0000 7f0e7ba33640 10 monclient: init 2024-03-29T11:35:01.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.011+0000 7f0e7ba33640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:01.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.011+0000 7f0e7ba33640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:35:01.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.011+0000 7f0e7ba33640 10 monclient(hunting): picked mon.noname-c con 0x7f0e74066b90 addr v1:172.21.15.72:6790/0 2024-03-29T11:35:01.022 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.011+0000 7f0e7ba33640 10 monclient(hunting): picked mon.noname-b con 0x7f0e7413f5e0 addr v1:172.21.15.89:6789/0 2024-03-29T11:35:01.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.011+0000 7f0e7ba33640 10 monclient(hunting): picked mon.noname-a con 0x7f0e741426a0 addr v1:172.21.15.72:6789/0 2024-03-29T11:35:01.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.011+0000 7f0e7ba33640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e7413c680 con 0x7f0e741426a0 2024-03-29T11:35:01.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.011+0000 7f0e7ba33640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e741d7f30 con 0x7f0e7413f5e0 2024-03-29T11:35:01.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.011+0000 7f0e7ba33640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e7406f500 con 0x7f0e74066b90 2024-03-29T11:35:01.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.011+0000 7f0e7ba33640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:01.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.011+0000 7f0e7ba33640 10 monclient(hunting): authenticate will time out at 873.284607s 2024-03-29T11:35:01.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.011+0000 7f0e78fce640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f0e7413f5e0 0x7f0e7413b360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57014/0 (socket says 172.21.15.89:57014) 2024-03-29T11:35:01.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.011+0000 7f0e78fce640 1 -- 172.21.15.89:0/3335508353 learned_addr learned my addr 172.21.15.89:0/3335508353 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-03-29T11:35:01.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.011+0000 7f0e69ffb640 10 client.?.objecter ms_handle_connect 0x7f0e7413f5e0 2024-03-29T11:35:01.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.011+0000 7f0e69ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:01.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.011+0000 7f0e69ffb640 10 client.?.objecter ms_handle_connect 0x7f0e74066b90 2024-03-29T11:35:01.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.011+0000 7f0e69ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:01.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.011+0000 7f0e69ffb640 10 client.?.objecter ms_handle_connect 0x7f0e741426a0 2024-03-29T11:35:01.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.011+0000 7f0e69ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:01.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.011+0000 7f0e69ffb640 1 -- 172.21.15.89:0/3335508353 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2451358892 0 0) 0x7f0e741d7f30 con 0x7f0e7413f5e0 2024-03-29T11:35:01.023 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.011+0000 7f0e69ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.011+0000 7f0e69ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.011+0000 7f0e69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.011+0000 7f0e69ffb640 10 monclient(hunting): my global_id is 4123 2024-03-29T11:35:01.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.011+0000 7f0e69ffb640 1 -- 172.21.15.89:0/3335508353 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e58004eb0 con 0x7f0e7413f5e0 2024-03-29T11:35:01.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.011+0000 7f0e69ffb640 1 -- 172.21.15.89:0/3335508353 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3163937686 0 0) 0x7f0e7406f500 con 0x7f0e74066b90 2024-03-29T11:35:01.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.011+0000 7f0e69ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.011+0000 7f0e69ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.011+0000 7f0e69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.011+0000 7f0e69ffb640 10 monclient(hunting): my global_id is 4112 2024-03-29T11:35:01.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.011+0000 7f0e69ffb640 1 -- 172.21.15.89:0/3335508353 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e741d7f30 con 0x7f0e74066b90 2024-03-29T11:35:01.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.011+0000 7f0e69ffb640 1 -- 172.21.15.89:0/3335508353 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3111707443 0 0) 0x7f0e7413c680 con 0x7f0e741426a0 2024-03-29T11:35:01.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.011+0000 7f0e69ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.011+0000 7f0e69ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.011+0000 7f0e69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.024 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.011+0000 7f0e69ffb640 10 monclient(hunting): my global_id is 4119 2024-03-29T11:35:01.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.011+0000 7f0e69ffb640 1 -- 172.21.15.89:0/3335508353 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e7406f500 con 0x7f0e741426a0 2024-03-29T11:35:01.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e69ffb640 1 -- 172.21.15.89:0/3335508353 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2604808592 0 0) 0x7f0e58004eb0 con 0x7f0e7413f5e0 2024-03-29T11:35:01.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e69ffb640 1 -- 172.21.15.89:0/3335508353 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e7413c680 con 0x7f0e7413f5e0 2024-03-29T11:35:01.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e69ffb640 1 -- 172.21.15.89:0/3335508353 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f0e70001b00 con 0x7f0e7413f5e0 2024-03-29T11:35:01.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e69ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e69ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:01.025 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e69ffb640 10 monclient(hunting): dump: 2024-03-29T11:35:01.025 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:01.025 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:01.025 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.025 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.025 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.025 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:01.025 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:01.025 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:01.026 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e69ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e69ffb640 1 -- 172.21.15.89:0/3335508353 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1629174405 0 0) 0x7f0e7406f500 con 0x7f0e741426a0 2024-03-29T11:35:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e69ffb640 1 -- 172.21.15.89:0/3335508353 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e58004eb0 con 0x7f0e741426a0 2024-03-29T11:35:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e69ffb640 1 -- 172.21.15.89:0/3335508353 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f0e6c001bd0 con 0x7f0e741426a0 2024-03-29T11:35:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e69ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e69ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e69ffb640 10 monclient(hunting): dump: 2024-03-29T11:35:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.026 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:01.027 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:01.027 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:01.027 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:01.027 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:01.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e69ffb640 1 -- 172.21.15.89:0/3335508353 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1049518628 0 0) 0x7f0e741d7f30 con 0x7f0e74066b90 2024-03-29T11:35:01.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e69ffb640 1 -- 172.21.15.89:0/3335508353 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e7406f500 con 0x7f0e74066b90 2024-03-29T11:35:01.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e69ffb640 1 -- 172.21.15.89:0/3335508353 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f0e64001a50 con 0x7f0e74066b90 2024-03-29T11:35:01.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e69ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e69ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:01.027 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e69ffb640 10 monclient(hunting): dump: 2024-03-29T11:35:01.027 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:01.027 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:01.027 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.027 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.027 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.027 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:01.028 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e69ffb640 1 -- 172.21.15.89:0/3335508353 <== mon.1 v1:172.21.15.89:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1034996114 0 0) 0x7f0e7413c680 con 0x7f0e7413f5e0 2024-03-29T11:35:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e69ffb640 1 -- 172.21.15.89:0/3335508353 >> v1:172.21.15.72:6790/0 conn(0x7f0e74066b90 legacy=0x7f0e741d7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e69ffb640 1 -- 172.21.15.89:0/3335508353 >> v1:172.21.15.72:6789/0 conn(0x7f0e741426a0 legacy=0x7f0e74070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e69ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:35:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e69ffb640 1 monclient: found mon.b 2024-03-29T11:35:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e69ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e69ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e69ffb640 1 -- 172.21.15.89:0/3335508353 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e7406f690 con 0x7f0e7413f5e0 2024-03-29T11:35:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e69ffb640 10 monclient: _finish_auth 0 2024-03-29T11:35:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e69ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:35:01.028 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e69ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:01.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e7ba33640 5 monclient: authenticate success, global_id 4123 2024-03-29T11:35:01.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e7ba33640 10 monclient: _renew_subs 2024-03-29T11:35:01.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e7ba33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:01.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e7ba33640 1 -- 172.21.15.89:0/3335508353 --> v1:172.21.15.89:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e741d7980 con 0x7f0e7413f5e0 2024-03-29T11:35:01.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e69ffb640 1 -- 172.21.15.89:0/3335508353 <== mon.1 v1:172.21.15.89:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e70004a90 con 0x7f0e7413f5e0 2024-03-29T11:35:01.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e69ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:01.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e69ffb640 1 -- 172.21.15.89:0/3335508353 <== mon.1 v1:172.21.15.89:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f0e70004da0 con 0x7f0e7413f5e0 2024-03-29T11:35:01.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e69ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e69ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:01.029 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e69ffb640 10 monclient: dump: 2024-03-29T11:35:01.029 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:01.029 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:01.029 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.029 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.029 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.029 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:01.030 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e69ffb640 1 -- 172.21.15.89:0/3335508353 <== mon.1 v1:172.21.15.89:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0e70005290 con 0x7f0e7413f5e0 2024-03-29T11:35:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e6b7fe640 4 set_mon_vals no callback set 2024-03-29T11:35:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e7ba33640 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e7ba33640 10 monclient: _renew_subs 2024-03-29T11:35:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e7ba33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e7ba33640 1 -- 172.21.15.89:0/3335508353 --> v1:172.21.15.89:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e7413bc50 con 0x7f0e7413f5e0 2024-03-29T11:35:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e7ba33640 1 librados: init done 2024-03-29T11:35:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e69ffb640 1 -- 172.21.15.89:0/3335508353 <== mon.1 v1:172.21.15.89:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 3313262649 0 0) 0x7f0e70005b70 con 0x7f0e7413f5e0 2024-03-29T11:35:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e69ffb640 10 client.4123.objecter ms_dispatch 0x7f0e74000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T11:35:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e69ffb640 3 client.4123.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T11:35:01.030 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e69ffb640 3 client.4123.objecter handle_osd_map decoding full epoch 3 2024-03-29T11:35:01.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.012+0000 7f0e69ffb640 20 client.4123.objecter dump_active .. 0 homeless 2024-03-29T11:35:01.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.013+0000 7f0e7ba33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:01.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.013+0000 7f0e7ba33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:01.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.013+0000 7f0e7ba33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:01.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.013+0000 7f0e7ba33640 1 -- 172.21.15.89:0/3335508353 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e740700b0 con 0x7f0e7413f5e0 2024-03-29T11:35:01.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.015+0000 7f0e69ffb640 1 -- 172.21.15.89:0/3335508353 <== mon.1 v1:172.21.15.89:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0e740700b0 con 0x7f0e7413f5e0 2024-03-29T11:35:01.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.015+0000 7f0e69ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:01.031 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.015+0000 7f0e69ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:01.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.087+0000 7f0e7ba33640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e054ff7c-3661-44b2-b369-5dbf77516540", "id": 1}] 2024-03-29T11:35:01.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.087+0000 7f0e7ba33640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e054ff7c-3661-44b2-b369-5dbf77516540", "id": 1}] 2024-03-29T11:35:01.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.087+0000 7f0e7ba33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:01.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.087+0000 7f0e7ba33640 1 -- 172.21.15.89:0/3335508353 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "osd new", "uuid": "e054ff7c-3661-44b2-b369-5dbf77516540", "id": 1} v 0) v1 -- 0x7f0e741d7d20 con 0x7f0e7413f5e0 2024-03-29T11:35:01.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.092+0000 7f0e69ffb640 1 -- 172.21.15.89:0/3335508353 <== mon.1 v1:172.21.15.89:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e054ff7c-3661-44b2-b369-5dbf77516540", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 4202657414 0 940483276) 0x7f0e70005d30 con 0x7f0e7413f5e0 2024-03-29T11:35:01.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.093+0000 7f0e69ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e054ff7c-3661-44b2-b369-5dbf77516540", "id": 1}] 2024-03-29T11:35:01.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.093+0000 7f0e69ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:01.091 INFO:teuthology.orchestra.run.smithi089.stdout:1 2024-03-29T11:35:01.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.093+0000 7f0e7ba33640 10 librados: watch_flush enter 2024-03-29T11:35:01.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.093+0000 7f0e7ba33640 10 librados: watch_flush exit 2024-03-29T11:35:01.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.093+0000 7f0e7ba33640 20 client.4123.objecter shutdown clearing up homeless session... 2024-03-29T11:35:01.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.093+0000 7f0e7ba33640 10 client.4123.objecter successfully canceled tick 2024-03-29T11:35:01.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.093+0000 7f0e7ba33640 10 monclient: shutdown 2024-03-29T11:35:01.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.093+0000 7f0e7ba33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:01.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.093+0000 7f0e7ba33640 1 -- 172.21.15.89:0/3335508353 >> v1:172.21.15.89:6789/0 conn(0x7f0e7413f5e0 legacy=0x7f0e7413b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.094+0000 7f0e7ba33640 1 -- 172.21.15.89:0/3335508353 shutdown_connections 2024-03-29T11:35:01.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.094+0000 7f0e7ba33640 1 -- 172.21.15.89:0/3335508353 >> 172.21.15.89:0/3335508353 conn(0x7f0e740b4a50 msgr2=0x7f0e741ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:01.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.094+0000 7f0e7ba33640 1 -- 172.21.15.89:0/3335508353 shutdown_connections 2024-03-29T11:35:01.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.094+0000 7f0e7ba33640 1 -- 172.21.15.89:0/3335508353 wait complete. 2024-03-29T11:35:01.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.094+0000 7f0e7ba33640 1 librados: shutdown 2024-03-29T11:35:01.102 DEBUG:teuthology.orchestra.run.smithi089:> sudo ceph --cluster ceph osd new fdaa9cde-4518-4a6b-a76f-526108e0ce08 2 2024-03-29T11:35:01.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.203+0000 7f4027754640 10 monclient: get_monmap_and_config 2024-03-29T11:35:01.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.203+0000 7f4027754640 10 monclient: build_initial_monmap 2024-03-29T11:35:01.201 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.203+0000 7f4027754640 10 monclient: monmap: 2024-03-29T11:35:01.201 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 0 2024-03-29T11:35:01.201 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:01.202 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:35:01.204835+0000 2024-03-29T11:35:01.202 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:35:01.204835+0000 2024-03-29T11:35:01.202 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:01.202 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:01.202 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:35:01.202 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:35:01.202 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:35:01.202 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:01.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.203+0000 7f4027754640 1 Processor -- start 2024-03-29T11:35:01.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.204+0000 7f4027754640 1 -- start start 2024-03-29T11:35:01.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.204+0000 7f4027754640 10 monclient: init 2024-03-29T11:35:01.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.204+0000 7f4027754640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:01.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.204+0000 7f4027754640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:35:01.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.204+0000 7f4027754640 10 monclient(hunting): picked mon.noname-c con 0x7f40200b1780 addr v1:172.21.15.89:6789/0 2024-03-29T11:35:01.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.204+0000 7f4027754640 10 monclient(hunting): picked mon.noname-a con 0x7f402013d7e0 addr v1:172.21.15.72:6789/0 2024-03-29T11:35:01.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.204+0000 7f4027754640 10 monclient(hunting): picked mon.noname-b con 0x7f4020143be0 addr v1:172.21.15.72:6790/0 2024-03-29T11:35:01.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.204+0000 7f4027754640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4020070b30 con 0x7f402013d7e0 2024-03-29T11:35:01.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.204+0000 7f4027754640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4020070da0 con 0x7f40200b1780 2024-03-29T11:35:01.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.204+0000 7f4027754640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f402009adb0 con 0x7f4020143be0 2024-03-29T11:35:01.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.204+0000 7f4027754640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:01.206 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.204+0000 7f4027754640 10 monclient(hunting): authenticate will time out at 873.477816s 2024-03-29T11:35:01.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.204+0000 7f40254f0640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f40200b1780 0x7f402013d0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57028/0 (socket says 172.21.15.89:57028) 2024-03-29T11:35:01.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.204+0000 7f40254f0640 1 -- 172.21.15.89:0/3906544559 learned_addr learned my addr 172.21.15.89:0/3906544559 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-03-29T11:35:01.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4017fff640 1 -- 172.21.15.89:0/3906544559 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2121341309 0 0) 0x7f4020070da0 con 0x7f40200b1780 2024-03-29T11:35:01.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4017fff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4017fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4017fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4017fff640 10 monclient(hunting): my global_id is 4126 2024-03-29T11:35:01.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4017fff640 1 -- 172.21.15.89:0/3906544559 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f400c003520 con 0x7f40200b1780 2024-03-29T11:35:01.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4017fff640 1 -- 172.21.15.89:0/3906544559 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2087460749 0 0) 0x7f4020070b30 con 0x7f402013d7e0 2024-03-29T11:35:01.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4017fff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4017fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4017fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4017fff640 10 monclient(hunting): my global_id is 4122 2024-03-29T11:35:01.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4017fff640 1 -- 172.21.15.89:0/3906544559 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4020070da0 con 0x7f402013d7e0 2024-03-29T11:35:01.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4017fff640 1 -- 172.21.15.89:0/3906544559 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3259675271 0 0) 0x7f402009adb0 con 0x7f4020143be0 2024-03-29T11:35:01.207 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4017fff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4017fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4017fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4017fff640 10 monclient(hunting): my global_id is 4115 2024-03-29T11:35:01.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4017fff640 1 -- 172.21.15.89:0/3906544559 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4020070b30 con 0x7f4020143be0 2024-03-29T11:35:01.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4017fff640 1 -- 172.21.15.89:0/3906544559 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 440151101 0 0) 0x7f400c003520 con 0x7f40200b1780 2024-03-29T11:35:01.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4017fff640 1 -- 172.21.15.89:0/3906544559 >> v1:172.21.15.72:6790/0 conn(0x7f4020143be0 legacy=0x7f402014b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4017fff640 1 -- 172.21.15.89:0/3906544559 >> v1:172.21.15.72:6789/0 conn(0x7f402013d7e0 legacy=0x7f40201434d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4017fff640 10 monclient: _finish_hunting 0 2024-03-29T11:35:01.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4017fff640 1 monclient: found mon.noname-c 2024-03-29T11:35:01.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4017fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:01.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4017fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.89:6789/0 2024-03-29T11:35:01.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4017fff640 1 -- 172.21.15.89:0/3906544559 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4020144070 con 0x7f40200b1780 2024-03-29T11:35:01.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4017fff640 10 monclient: _finish_auth 0 2024-03-29T11:35:01.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4017fff640 10 monclient: _check_auth_tickets 2024-03-29T11:35:01.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4017fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:01.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4027754640 5 monclient: authenticate success, global_id 4126 2024-03-29T11:35:01.208 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4017fff640 1 -- 172.21.15.89:0/3906544559 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f4010001690 con 0x7f40200b1780 2024-03-29T11:35:01.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4027754640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:01.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4017fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4017fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:01.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4017fff640 10 monclient: dump: 2024-03-29T11:35:01.209 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:01.209 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:01.209 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.209 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.209 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.209 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:01.209 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:01.209 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:01.209 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:01.209 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:01.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4027754640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:01.209 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4017fff640 1 -- 172.21.15.89:0/3906544559 <== mon.1 v1:172.21.15.89:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4010001870 con 0x7f40200b1780 2024-03-29T11:35:01.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4017fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:01.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4027754640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:01.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4017fff640 1 -- 172.21.15.89:0/3906544559 <== mon.1 v1:172.21.15.89:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f4010001cc0 con 0x7f40200b1780 2024-03-29T11:35:01.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4027754640 4 set_mon_vals no callback set 2024-03-29T11:35:01.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4017fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4017fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:01.210 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.205+0000 7f4017fff640 10 monclient: dump: 2024-03-29T11:35:01.210 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:01.210 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:01.210 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.210 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.210 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.210 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:01.210 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:01.210 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:01.211 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:01.211 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:01.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.206+0000 7f4027754640 10 monclient: shutdown 2024-03-29T11:35:01.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.206+0000 7f4027754640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:01.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.206+0000 7f4027754640 1 -- 172.21.15.89:0/3906544559 >> v1:172.21.15.89:6789/0 conn(0x7f40200b1780 legacy=0x7f402013d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.206+0000 7f4027754640 1 -- 172.21.15.89:0/3906544559 shutdown_connections 2024-03-29T11:35:01.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.206+0000 7f4027754640 1 -- 172.21.15.89:0/3906544559 >> 172.21.15.89:0/3906544559 conn(0x7f40200b58f0 msgr2=0x7f40200b5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:01.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.206+0000 7f4027754640 1 -- 172.21.15.89:0/3906544559 shutdown_connections 2024-03-29T11:35:01.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.206+0000 7f4027754640 1 -- 172.21.15.89:0/3906544559 wait complete. 2024-03-29T11:35:01.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.206+0000 7f4027754640 10 monclient: build_initial_monmap 2024-03-29T11:35:01.211 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.206+0000 7f4027754640 10 monclient: monmap: 2024-03-29T11:35:01.211 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 0 2024-03-29T11:35:01.211 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:01.211 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 0.000000 2024-03-29T11:35:01.211 INFO:teuthology.orchestra.run.smithi089.stderr:created 0.000000 2024-03-29T11:35:01.211 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:01.211 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:01.212 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:35:01.212 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:35:01.212 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:35:01.212 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:01.212 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-29T11:35:01.207+0000 7f1f7fc2b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:35:01.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.207+0000 7f4027754640 1 librados: starting msgr at 2024-03-29T11:35:01.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.207+0000 7f4027754640 1 librados: starting objecter 2024-03-29T11:35:01.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.207+0000 7f4027754640 1 Processor -- start 2024-03-29T11:35:01.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.207+0000 7f4027754640 1 -- start start 2024-03-29T11:35:01.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.207+0000 7f4027754640 1 librados: setting wanted keys 2024-03-29T11:35:01.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.207+0000 7f4027754640 1 librados: calling monclient init 2024-03-29T11:35:01.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.207+0000 7f4027754640 10 monclient: init 2024-03-29T11:35:01.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.207+0000 7f4027754640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:01.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.207+0000 7f4027754640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:35:01.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.207+0000 7f4027754640 10 monclient(hunting): picked mon.noname-a con 0x7f40200b1780 addr v1:172.21.15.72:6789/0 2024-03-29T11:35:01.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.207+0000 7f4027754640 10 monclient(hunting): picked mon.noname-b con 0x7f402013d7e0 addr v1:172.21.15.89:6789/0 2024-03-29T11:35:01.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.207+0000 7f4027754640 10 monclient(hunting): picked mon.noname-c con 0x7f4020143be0 addr v1:172.21.15.72:6790/0 2024-03-29T11:35:01.218 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.207+0000 7f4027754640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4020141e00 con 0x7f40200b1780 2024-03-29T11:35:01.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.207+0000 7f4027754640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4020133440 con 0x7f402013d7e0 2024-03-29T11:35:01.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.207+0000 7f4027754640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4020142370 con 0x7f4020143be0 2024-03-29T11:35:01.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.207+0000 7f4027754640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:01.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.207+0000 7f4027754640 10 monclient(hunting): authenticate will time out at 873.480882s 2024-03-29T11:35:01.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.207+0000 7f4024cef640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f402013d7e0 0x7f4020132620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57044/0 (socket says 172.21.15.89:57044) 2024-03-29T11:35:01.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.207+0000 7f4024cef640 1 -- 172.21.15.89:0/1896682764 learned_addr learned my addr 172.21.15.89:0/1896682764 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-03-29T11:35:01.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 10 client.?.objecter ms_handle_connect 0x7f402013d7e0 2024-03-29T11:35:01.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:01.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 10 client.?.objecter ms_handle_connect 0x7f40200b1780 2024-03-29T11:35:01.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:01.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 10 client.?.objecter ms_handle_connect 0x7f4020143be0 2024-03-29T11:35:01.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:01.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 1 -- 172.21.15.89:0/1896682764 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 748875586 0 0) 0x7f4020133440 con 0x7f402013d7e0 2024-03-29T11:35:01.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.219 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 10 monclient(hunting): my global_id is 4129 2024-03-29T11:35:01.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 1 -- 172.21.15.89:0/1896682764 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4004004eb0 con 0x7f402013d7e0 2024-03-29T11:35:01.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 1 -- 172.21.15.89:0/1896682764 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 624985488 0 0) 0x7f4020141e00 con 0x7f40200b1780 2024-03-29T11:35:01.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 10 monclient(hunting): my global_id is 4125 2024-03-29T11:35:01.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 1 -- 172.21.15.89:0/1896682764 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4020133440 con 0x7f40200b1780 2024-03-29T11:35:01.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 1 -- 172.21.15.89:0/1896682764 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4061363607 0 0) 0x7f4020142370 con 0x7f4020143be0 2024-03-29T11:35:01.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 10 monclient(hunting): my global_id is 4118 2024-03-29T11:35:01.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 1 -- 172.21.15.89:0/1896682764 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4020141e00 con 0x7f4020143be0 2024-03-29T11:35:01.220 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 1 -- 172.21.15.89:0/1896682764 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3238143058 0 0) 0x7f4004004eb0 con 0x7f402013d7e0 2024-03-29T11:35:01.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 1 -- 172.21.15.89:0/1896682764 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4020142370 con 0x7f402013d7e0 2024-03-29T11:35:01.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 1 -- 172.21.15.89:0/1896682764 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f401c0012d0 con 0x7f402013d7e0 2024-03-29T11:35:01.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:01.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 10 monclient(hunting): dump: 2024-03-29T11:35:01.221 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:01.221 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:01.221 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.221 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.221 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.221 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:01.221 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:01.221 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:01.221 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:01.221 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:01.221 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:01.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 1 -- 172.21.15.89:0/1896682764 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3687476061 0 0) 0x7f4020133440 con 0x7f40200b1780 2024-03-29T11:35:01.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 1 -- 172.21.15.89:0/1896682764 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4004004eb0 con 0x7f40200b1780 2024-03-29T11:35:01.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 1 -- 172.21.15.89:0/1896682764 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3764117518 0 0) 0x7f4020141e00 con 0x7f4020143be0 2024-03-29T11:35:01.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 1 -- 172.21.15.89:0/1896682764 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4020133440 con 0x7f4020143be0 2024-03-29T11:35:01.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 1 -- 172.21.15.89:0/1896682764 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f4010001b80 con 0x7f40200b1780 2024-03-29T11:35:01.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:01.222 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 10 monclient(hunting): dump: 2024-03-29T11:35:01.222 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:01.222 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:01.222 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.222 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.222 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.222 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:01.222 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:01.222 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:01.223 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:01.223 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:01.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 1 -- 172.21.15.89:0/1896682764 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f4018001b50 con 0x7f4020143be0 2024-03-29T11:35:01.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:01.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 10 monclient(hunting): dump: 2024-03-29T11:35:01.223 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:01.223 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:01.223 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.223 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.223 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.223 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:01.223 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:01.223 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:01.223 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:01.223 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:01.223 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 1 -- 172.21.15.89:0/1896682764 <== mon.1 v1:172.21.15.89:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3574909871 0 0) 0x7f4020142370 con 0x7f402013d7e0 2024-03-29T11:35:01.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 1 -- 172.21.15.89:0/1896682764 >> v1:172.21.15.72:6790/0 conn(0x7f4020143be0 legacy=0x7f4020132d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 1 -- 172.21.15.89:0/1896682764 >> v1:172.21.15.72:6789/0 conn(0x7f40200b1780 legacy=0x7f40201da4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:35:01.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 1 monclient: found mon.b 2024-03-29T11:35:01.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:01.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:01.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 1 -- 172.21.15.89:0/1896682764 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f402006c940 con 0x7f402013d7e0 2024-03-29T11:35:01.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 10 monclient: _finish_auth 0 2024-03-29T11:35:01.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:35:01.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:01.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4027754640 5 monclient: authenticate success, global_id 4129 2024-03-29T11:35:01.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4027754640 10 monclient: _renew_subs 2024-03-29T11:35:01.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4027754640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:01.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4027754640 1 -- 172.21.15.89:0/1896682764 --> v1:172.21.15.89:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40200af360 con 0x7f402013d7e0 2024-03-29T11:35:01.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 1 -- 172.21.15.89:0/1896682764 <== mon.1 v1:172.21.15.89:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f401c001450 con 0x7f402013d7e0 2024-03-29T11:35:01.224 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:01.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 1 -- 172.21.15.89:0/1896682764 <== mon.1 v1:172.21.15.89:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f401c0038a0 con 0x7f402013d7e0 2024-03-29T11:35:01.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:01.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 10 monclient: dump: 2024-03-29T11:35:01.225 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:01.225 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:01.225 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.225 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.225 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.225 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:01.225 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:01.225 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:01.225 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:01.225 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:01.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f40177fe640 4 set_mon_vals no callback set 2024-03-29T11:35:01.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4027754640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:01.225 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4027754640 10 monclient: _renew_subs 2024-03-29T11:35:01.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4027754640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:01.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4027754640 1 -- 172.21.15.89:0/1896682764 --> v1:172.21.15.89:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f40201418e0 con 0x7f402013d7e0 2024-03-29T11:35:01.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4015ffb640 1 -- 172.21.15.89:0/1896682764 <== mon.1 v1:172.21.15.89:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f401c003e10 con 0x7f402013d7e0 2024-03-29T11:35:01.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.208+0000 7f4027754640 1 librados: init done 2024-03-29T11:35:01.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.209+0000 7f4015ffb640 1 -- 172.21.15.89:0/1896682764 <== mon.1 v1:172.21.15.89:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 1390753601 0 0) 0x7f401c005780 con 0x7f402013d7e0 2024-03-29T11:35:01.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.209+0000 7f4015ffb640 10 client.4129.objecter ms_dispatch 0x7f4020000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T11:35:01.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.209+0000 7f4015ffb640 3 client.4129.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T11:35:01.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.209+0000 7f4015ffb640 3 client.4129.objecter handle_osd_map decoding full epoch 4 2024-03-29T11:35:01.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.209+0000 7f4015ffb640 20 client.4129.objecter dump_active .. 0 homeless 2024-03-29T11:35:01.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.209+0000 7f4027754640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:01.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.209+0000 7f4027754640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:01.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.209+0000 7f4027754640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:01.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.209+0000 7f4027754640 1 -- 172.21.15.89:0/1896682764 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4020067c00 con 0x7f402013d7e0 2024-03-29T11:35:01.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.211+0000 7f4015ffb640 1 -- 172.21.15.89:0/1896682764 <== mon.1 v1:172.21.15.89:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f401c005940 con 0x7f402013d7e0 2024-03-29T11:35:01.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.211+0000 7f4015ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:01.226 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.211+0000 7f4015ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:01.241 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-29T11:35:01.243+0000 7fc853e3e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:35:01.261 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-29T11:35:01.263+0000 7f1f7fc2b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:35:01.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.284+0000 7f4027754640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "fdaa9cde-4518-4a6b-a76f-526108e0ce08", "id": 2}] 2024-03-29T11:35:01.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.284+0000 7f4027754640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "fdaa9cde-4518-4a6b-a76f-526108e0ce08", "id": 2}] 2024-03-29T11:35:01.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.284+0000 7f4027754640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:01.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.284+0000 7f4027754640 1 -- 172.21.15.89:0/1896682764 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "osd new", "uuid": "fdaa9cde-4518-4a6b-a76f-526108e0ce08", "id": 2} v 0) v1 -- 0x7f40200af700 con 0x7f402013d7e0 2024-03-29T11:35:01.289 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.291+0000 7f4015ffb640 1 -- 172.21.15.89:0/1896682764 <== mon.1 v1:172.21.15.89:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fdaa9cde-4518-4a6b-a76f-526108e0ce08", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 1661060560 0 216599125) 0x7f401c005c90 con 0x7f402013d7e0 2024-03-29T11:35:01.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.291+0000 7f4015ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "fdaa9cde-4518-4a6b-a76f-526108e0ce08", "id": 2}] 2024-03-29T11:35:01.290 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.291+0000 7f4015ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:01.290 INFO:teuthology.orchestra.run.smithi089.stdout:2 2024-03-29T11:35:01.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.292+0000 7f4027754640 10 librados: watch_flush enter 2024-03-29T11:35:01.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.292+0000 7f4027754640 10 librados: watch_flush exit 2024-03-29T11:35:01.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.292+0000 7f4027754640 20 client.4129.objecter shutdown clearing up homeless session... 2024-03-29T11:35:01.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.292+0000 7f4027754640 10 client.4129.objecter successfully canceled tick 2024-03-29T11:35:01.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.292+0000 7f4027754640 10 monclient: shutdown 2024-03-29T11:35:01.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.292+0000 7f4027754640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:01.291 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.292+0000 7f4027754640 1 -- 172.21.15.89:0/1896682764 >> v1:172.21.15.89:6789/0 conn(0x7f402013d7e0 legacy=0x7f4020132620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.292+0000 7f4027754640 1 -- 172.21.15.89:0/1896682764 shutdown_connections 2024-03-29T11:35:01.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.292+0000 7f4027754640 1 -- 172.21.15.89:0/1896682764 >> 172.21.15.89:0/1896682764 conn(0x7f40200b58f0 msgr2=0x7f4020069bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:01.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.292+0000 7f4027754640 1 -- 172.21.15.89:0/1896682764 shutdown_connections 2024-03-29T11:35:01.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.292+0000 7f4027754640 1 -- 172.21.15.89:0/1896682764 wait complete. 2024-03-29T11:35:01.292 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.292+0000 7f4027754640 1 librados: shutdown 2024-03-29T11:35:01.295 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-29T11:35:01.297+0000 7fc853e3e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:35:01.301 DEBUG:teuthology.orchestra.run.smithi089:> sudo ceph --cluster ceph osd new 1497998c-9d8a-47c5-ad4d-21f1a96c41b3 3 2024-03-29T11:35:01.309 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-29T11:35:01.310+0000 7f1f7fc2b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:35:01.339 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-29T11:35:01.341+0000 7fc853e3e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:35:01.365 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-29T11:35:01.367+0000 7f1f7fc2b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:35:01.391 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-29T11:35:01.394+0000 7fc853e3e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:35:01.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.402+0000 7fb54545c640 10 monclient: get_monmap_and_config 2024-03-29T11:35:01.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.402+0000 7fb54545c640 10 monclient: build_initial_monmap 2024-03-29T11:35:01.401 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.402+0000 7fb54545c640 10 monclient: monmap: 2024-03-29T11:35:01.401 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 0 2024-03-29T11:35:01.401 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:01.401 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:35:01.403973+0000 2024-03-29T11:35:01.402 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:35:01.403973+0000 2024-03-29T11:35:01.402 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:01.402 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:01.402 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:35:01.402 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:35:01.402 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:35:01.402 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:01.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.403+0000 7fb54545c640 1 Processor -- start 2024-03-29T11:35:01.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.403+0000 7fb54545c640 1 -- start start 2024-03-29T11:35:01.402 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.403+0000 7fb54545c640 10 monclient: init 2024-03-29T11:35:01.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.403+0000 7fb54545c640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:01.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.403+0000 7fb54545c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:35:01.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.403+0000 7fb54545c640 10 monclient(hunting): picked mon.noname-c con 0x7fb5400b8240 addr v1:172.21.15.89:6789/0 2024-03-29T11:35:01.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.403+0000 7fb54545c640 10 monclient(hunting): picked mon.noname-b con 0x7fb5400b1f60 addr v1:172.21.15.72:6790/0 2024-03-29T11:35:01.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.403+0000 7fb54545c640 10 monclient(hunting): picked mon.noname-a con 0x7fb54009c8c0 addr v1:172.21.15.72:6789/0 2024-03-29T11:35:01.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.403+0000 7fb54545c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb54009bd80 con 0x7fb54009c8c0 2024-03-29T11:35:01.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.403+0000 7fb54545c640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb54009b960 con 0x7fb5400b8240 2024-03-29T11:35:01.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.403+0000 7fb54545c640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5400b2ec0 con 0x7fb5400b1f60 2024-03-29T11:35:01.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.403+0000 7fb54545c640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:01.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.403+0000 7fb54545c640 10 monclient(hunting): authenticate will time out at 873.676779s 2024-03-29T11:35:01.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.403+0000 7fb53effd640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb5400b8240 0x7fb54013f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57046/0 (socket says 172.21.15.89:57046) 2024-03-29T11:35:01.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.403+0000 7fb53effd640 1 -- 172.21.15.89:0/3583931639 learned_addr learned my addr 172.21.15.89:0/3583931639 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-03-29T11:35:01.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb53dffb640 1 -- 172.21.15.89:0/3583931639 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1895089973 0 0) 0x7fb54009b960 con 0x7fb5400b8240 2024-03-29T11:35:01.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb53dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb53dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb53dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb53dffb640 10 monclient(hunting): my global_id is 4132 2024-03-29T11:35:01.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb53dffb640 1 -- 172.21.15.89:0/3583931639 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb528003520 con 0x7fb5400b8240 2024-03-29T11:35:01.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb53dffb640 1 -- 172.21.15.89:0/3583931639 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3358773217 0 0) 0x7fb54009bd80 con 0x7fb54009c8c0 2024-03-29T11:35:01.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb53dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb53dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb53dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb53dffb640 10 monclient(hunting): my global_id is 4128 2024-03-29T11:35:01.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb53dffb640 1 -- 172.21.15.89:0/3583931639 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb54009b960 con 0x7fb54009c8c0 2024-03-29T11:35:01.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb53dffb640 1 -- 172.21.15.89:0/3583931639 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1658911308 0 0) 0x7fb5400b2ec0 con 0x7fb5400b1f60 2024-03-29T11:35:01.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb53dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb53dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb53dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.415 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb53dffb640 10 monclient(hunting): my global_id is 4121 2024-03-29T11:35:01.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb53dffb640 1 -- 172.21.15.89:0/3583931639 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb54009bd80 con 0x7fb5400b1f60 2024-03-29T11:35:01.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb53dffb640 1 -- 172.21.15.89:0/3583931639 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1677060859 0 0) 0x7fb528003520 con 0x7fb5400b8240 2024-03-29T11:35:01.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb53dffb640 1 -- 172.21.15.89:0/3583931639 >> v1:172.21.15.72:6790/0 conn(0x7fb5400b1f60 legacy=0x7fb54009c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb53dffb640 1 -- 172.21.15.89:0/3583931639 >> v1:172.21.15.72:6789/0 conn(0x7fb54009c8c0 legacy=0x7fb54014d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb53dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:35:01.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb53dffb640 1 monclient: found mon.noname-c 2024-03-29T11:35:01.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb53dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:01.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb53dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.89:6789/0 2024-03-29T11:35:01.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb53dffb640 1 -- 172.21.15.89:0/3583931639 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb54013fb60 con 0x7fb5400b8240 2024-03-29T11:35:01.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb53dffb640 10 monclient: _finish_auth 0 2024-03-29T11:35:01.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb53dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:35:01.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb53dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:01.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb53dffb640 1 -- 172.21.15.89:0/3583931639 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fb530001690 con 0x7fb5400b8240 2024-03-29T11:35:01.416 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb54545c640 5 monclient: authenticate success, global_id 4132 2024-03-29T11:35:01.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb54545c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:01.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb53dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb53dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:01.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb53dffb640 10 monclient: dump: 2024-03-29T11:35:01.417 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:01.417 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:01.417 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.417 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.417 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.417 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:01.417 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:01.417 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:01.417 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:01.417 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:01.417 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb54545c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:01.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb53dffb640 1 -- 172.21.15.89:0/3583931639 <== mon.1 v1:172.21.15.89:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb530001870 con 0x7fb5400b8240 2024-03-29T11:35:01.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb53dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:01.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb54545c640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:01.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb53dffb640 1 -- 172.21.15.89:0/3583931639 <== mon.1 v1:172.21.15.89:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fb530001c80 con 0x7fb5400b8240 2024-03-29T11:35:01.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb54545c640 4 set_mon_vals no callback set 2024-03-29T11:35:01.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb53dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb53dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:01.418 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.404+0000 7fb53dffb640 10 monclient: dump: 2024-03-29T11:35:01.418 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:01.418 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:01.418 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.418 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.418 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.418 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:01.419 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:01.419 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:01.419 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:01.419 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:01.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.405+0000 7fb54545c640 10 monclient: shutdown 2024-03-29T11:35:01.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.405+0000 7fb54545c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:01.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.405+0000 7fb54545c640 1 -- 172.21.15.89:0/3583931639 >> v1:172.21.15.89:6789/0 conn(0x7fb5400b8240 legacy=0x7fb54013f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.405+0000 7fb54545c640 1 -- 172.21.15.89:0/3583931639 shutdown_connections 2024-03-29T11:35:01.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.405+0000 7fb54545c640 1 -- 172.21.15.89:0/3583931639 >> 172.21.15.89:0/3583931639 conn(0x7fb540066b00 msgr2=0x7fb540066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:01.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.405+0000 7fb54545c640 1 -- 172.21.15.89:0/3583931639 shutdown_connections 2024-03-29T11:35:01.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.405+0000 7fb54545c640 1 -- 172.21.15.89:0/3583931639 wait complete. 2024-03-29T11:35:01.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.405+0000 7fb54545c640 10 monclient: build_initial_monmap 2024-03-29T11:35:01.419 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.405+0000 7fb54545c640 10 monclient: monmap: 2024-03-29T11:35:01.419 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 0 2024-03-29T11:35:01.420 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:01.420 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 0.000000 2024-03-29T11:35:01.420 INFO:teuthology.orchestra.run.smithi089.stderr:created 0.000000 2024-03-29T11:35:01.420 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:01.420 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:01.420 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:35:01.420 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:35:01.420 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:35:01.420 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:01.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.405+0000 7fb54545c640 1 librados: starting msgr at 2024-03-29T11:35:01.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.405+0000 7fb54545c640 1 librados: starting objecter 2024-03-29T11:35:01.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.406+0000 7fb54545c640 1 Processor -- start 2024-03-29T11:35:01.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.406+0000 7fb54545c640 1 -- start start 2024-03-29T11:35:01.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.406+0000 7fb54545c640 1 librados: setting wanted keys 2024-03-29T11:35:01.420 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.406+0000 7fb54545c640 1 librados: calling monclient init 2024-03-29T11:35:01.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.406+0000 7fb54545c640 10 monclient: init 2024-03-29T11:35:01.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.406+0000 7fb54545c640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:01.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.406+0000 7fb54545c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:35:01.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.406+0000 7fb54545c640 10 monclient(hunting): picked mon.noname-a con 0x7fb54009c8c0 addr v1:172.21.15.72:6789/0 2024-03-29T11:35:01.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.406+0000 7fb54545c640 10 monclient(hunting): picked mon.noname-c con 0x7fb5400b8240 addr v1:172.21.15.72:6790/0 2024-03-29T11:35:01.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.406+0000 7fb54545c640 10 monclient(hunting): picked mon.noname-b con 0x7fb5401d7ac0 addr v1:172.21.15.89:6789/0 2024-03-29T11:35:01.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.406+0000 7fb54545c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb540135830 con 0x7fb54009c8c0 2024-03-29T11:35:01.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.406+0000 7fb54545c640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5400b08d0 con 0x7fb5401d7ac0 2024-03-29T11:35:01.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.406+0000 7fb54545c640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5400af1b0 con 0x7fb5400b8240 2024-03-29T11:35:01.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.406+0000 7fb54545c640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:01.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.406+0000 7fb54545c640 10 monclient(hunting): authenticate will time out at 873.679804s 2024-03-29T11:35:01.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.406+0000 7fb53f7fe640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb5401d7ac0 0x7fb540135120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57052/0 (socket says 172.21.15.89:57052) 2024-03-29T11:35:01.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.406+0000 7fb53f7fe640 1 -- 172.21.15.89:0/3745865709 learned_addr learned my addr 172.21.15.89:0/3745865709 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-03-29T11:35:01.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.406+0000 7fb5277fe640 10 client.?.objecter ms_handle_connect 0x7fb5401d7ac0 2024-03-29T11:35:01.421 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.406+0000 7fb5277fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:01.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 10 client.?.objecter ms_handle_connect 0x7fb54009c8c0 2024-03-29T11:35:01.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:01.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 10 client.?.objecter ms_handle_connect 0x7fb5400b8240 2024-03-29T11:35:01.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:01.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 1 -- 172.21.15.89:0/3745865709 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1130647029 0 0) 0x7fb5400b08d0 con 0x7fb5401d7ac0 2024-03-29T11:35:01.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 10 monclient(hunting): my global_id is 4135 2024-03-29T11:35:01.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 1 -- 172.21.15.89:0/3745865709 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb52c004eb0 con 0x7fb5401d7ac0 2024-03-29T11:35:01.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 1 -- 172.21.15.89:0/3745865709 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 611355207 0 0) 0x7fb540135830 con 0x7fb54009c8c0 2024-03-29T11:35:01.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.422 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 10 monclient(hunting): my global_id is 4131 2024-03-29T11:35:01.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 1 -- 172.21.15.89:0/3745865709 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5400b08d0 con 0x7fb54009c8c0 2024-03-29T11:35:01.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 1 -- 172.21.15.89:0/3745865709 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1497612703 0 0) 0x7fb5400af1b0 con 0x7fb5400b8240 2024-03-29T11:35:01.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 10 monclient(hunting): my global_id is 4124 2024-03-29T11:35:01.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 1 -- 172.21.15.89:0/3745865709 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb540135830 con 0x7fb5400b8240 2024-03-29T11:35:01.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 1 -- 172.21.15.89:0/3745865709 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4005027977 0 0) 0x7fb52c004eb0 con 0x7fb5401d7ac0 2024-03-29T11:35:01.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 1 -- 172.21.15.89:0/3745865709 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5400af1b0 con 0x7fb5401d7ac0 2024-03-29T11:35:01.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 1 -- 172.21.15.89:0/3745865709 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fb538001cb0 con 0x7fb5401d7ac0 2024-03-29T11:35:01.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:01.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 10 monclient(hunting): dump: 2024-03-29T11:35:01.423 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:01.423 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:01.424 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.424 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.424 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.424 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:01.424 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:01.424 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:01.424 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:01.424 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:01.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:01.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 1 -- 172.21.15.89:0/3745865709 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2879704511 0 0) 0x7fb5400b08d0 con 0x7fb54009c8c0 2024-03-29T11:35:01.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 1 -- 172.21.15.89:0/3745865709 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb52c004eb0 con 0x7fb54009c8c0 2024-03-29T11:35:01.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 1 -- 172.21.15.89:0/3745865709 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fb530001bc0 con 0x7fb54009c8c0 2024-03-29T11:35:01.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:01.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 10 monclient(hunting): dump: 2024-03-29T11:35:01.424 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:01.425 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:01.425 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.425 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.425 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.425 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:01.425 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:01.425 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:01.425 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:01.425 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:01.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 1 -- 172.21.15.89:0/3745865709 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2944159125 0 0) 0x7fb540135830 con 0x7fb5400b8240 2024-03-29T11:35:01.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 1 -- 172.21.15.89:0/3745865709 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5400b08d0 con 0x7fb5400b8240 2024-03-29T11:35:01.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 1 -- 172.21.15.89:0/3745865709 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fb534001830 con 0x7fb5400b8240 2024-03-29T11:35:01.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:01.425 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 10 monclient(hunting): dump: 2024-03-29T11:35:01.426 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:01.426 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:01.426 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.426 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.426 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.426 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:01.426 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:01.426 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:01.426 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:01.426 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:01.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 1 -- 172.21.15.89:0/3745865709 <== mon.1 v1:172.21.15.89:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 576614666 0 0) 0x7fb5400af1b0 con 0x7fb5401d7ac0 2024-03-29T11:35:01.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 1 -- 172.21.15.89:0/3745865709 >> v1:172.21.15.72:6790/0 conn(0x7fb5400b8240 legacy=0x7fb540134a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 1 -- 172.21.15.89:0/3745865709 >> v1:172.21.15.72:6789/0 conn(0x7fb54009c8c0 legacy=0x7fb5401d73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 10 monclient: _finish_hunting 0 2024-03-29T11:35:01.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 1 monclient: found mon.b 2024-03-29T11:35:01.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:01.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:01.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 1 -- 172.21.15.89:0/3745865709 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5400af6d0 con 0x7fb5401d7ac0 2024-03-29T11:35:01.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 10 monclient: _finish_auth 0 2024-03-29T11:35:01.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 10 monclient: _check_auth_tickets 2024-03-29T11:35:01.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:01.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb54545c640 5 monclient: authenticate success, global_id 4135 2024-03-29T11:35:01.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb54545c640 10 monclient: _renew_subs 2024-03-29T11:35:01.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb54545c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:01.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb54545c640 1 -- 172.21.15.89:0/3745865709 --> v1:172.21.15.89:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5400aedd0 con 0x7fb5401d7ac0 2024-03-29T11:35:01.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb54545c640 10 client.4135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:01.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb54545c640 10 monclient: _renew_subs 2024-03-29T11:35:01.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb54545c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:01.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb54545c640 1 -- 172.21.15.89:0/3745865709 --> v1:172.21.15.89:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5400aea50 con 0x7fb5401d7ac0 2024-03-29T11:35:01.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb54545c640 1 librados: init done 2024-03-29T11:35:01.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 1 -- 172.21.15.89:0/3745865709 <== mon.1 v1:172.21.15.89:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb538003990 con 0x7fb5401d7ac0 2024-03-29T11:35:01.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:01.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 1 -- 172.21.15.89:0/3745865709 <== mon.1 v1:172.21.15.89:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fb538003da0 con 0x7fb5401d7ac0 2024-03-29T11:35:01.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:01.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb53d7fa640 4 set_mon_vals no callback set 2024-03-29T11:35:01.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 10 monclient: dump: 2024-03-29T11:35:01.428 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:01.429 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:01.429 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.429 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.429 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.429 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:01.429 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:01.429 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:01.429 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:01.429 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:01.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 1 -- 172.21.15.89:0/3745865709 <== mon.1 v1:172.21.15.89:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb538004290 con 0x7fb5401d7ac0 2024-03-29T11:35:01.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 1 -- 172.21.15.89:0/3745865709 <== mon.1 v1:172.21.15.89:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 3476488540 0 0) 0x7fb538004540 con 0x7fb5401d7ac0 2024-03-29T11:35:01.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 10 client.4135.objecter ms_dispatch 0x7fb540000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T11:35:01.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 3 client.4135.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T11:35:01.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 3 client.4135.objecter handle_osd_map decoding full epoch 5 2024-03-29T11:35:01.429 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.407+0000 7fb5277fe640 20 client.4135.objecter dump_active .. 0 homeless 2024-03-29T11:35:01.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.408+0000 7fb54545c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:01.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.408+0000 7fb54545c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:01.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.408+0000 7fb54545c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:01.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.408+0000 7fb54545c640 1 -- 172.21.15.89:0/3745865709 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb520005080 con 0x7fb5401d7ac0 2024-03-29T11:35:01.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.410+0000 7fb5277fe640 1 -- 172.21.15.89:0/3745865709 <== mon.1 v1:172.21.15.89:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb538005b40 con 0x7fb5401d7ac0 2024-03-29T11:35:01.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.410+0000 7fb5277fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:01.430 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.410+0000 7fb5277fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:01.430 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-29T11:35:01.412+0000 7f1f7fc2b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:35:01.434 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-29T11:35:01.437+0000 7fc853e3e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:35:01.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.483+0000 7fb54545c640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "1497998c-9d8a-47c5-ad4d-21f1a96c41b3", "id": 3}] 2024-03-29T11:35:01.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.483+0000 7fb54545c640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "1497998c-9d8a-47c5-ad4d-21f1a96c41b3", "id": 3}] 2024-03-29T11:35:01.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.483+0000 7fb54545c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:01.481 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.483+0000 7fb54545c640 1 -- 172.21.15.89:0/3745865709 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "osd new", "uuid": "1497998c-9d8a-47c5-ad4d-21f1a96c41b3", "id": 3} v 0) v1 -- 0x7fb5200058b0 con 0x7fb5401d7ac0 2024-03-29T11:35:01.488 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.490+0000 7fb5277fe640 1 -- 172.21.15.89:0/3745865709 <== mon.1 v1:172.21.15.89:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1497998c-9d8a-47c5-ad4d-21f1a96c41b3", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 378323579 0 525046306) 0x7fb538005e20 con 0x7fb5401d7ac0 2024-03-29T11:35:01.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.490+0000 7fb5277fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "1497998c-9d8a-47c5-ad4d-21f1a96c41b3", "id": 3}] 2024-03-29T11:35:01.489 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.490+0000 7fb5277fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:01.489 INFO:teuthology.orchestra.run.smithi089.stdout:3 2024-03-29T11:35:01.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.491+0000 7fb54545c640 10 librados: watch_flush enter 2024-03-29T11:35:01.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.491+0000 7fb54545c640 10 librados: watch_flush exit 2024-03-29T11:35:01.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.491+0000 7fb54545c640 20 client.4135.objecter shutdown clearing up homeless session... 2024-03-29T11:35:01.490 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.491+0000 7fb54545c640 10 client.4135.objecter successfully canceled tick 2024-03-29T11:35:01.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.491+0000 7fb54545c640 10 monclient: shutdown 2024-03-29T11:35:01.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.491+0000 7fb54545c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:01.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.491+0000 7fb54545c640 1 -- 172.21.15.89:0/3745865709 >> v1:172.21.15.89:6789/0 conn(0x7fb5401d7ac0 legacy=0x7fb540135120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.491+0000 7fb54545c640 1 -- 172.21.15.89:0/3745865709 shutdown_connections 2024-03-29T11:35:01.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.491+0000 7fb54545c640 1 -- 172.21.15.89:0/3745865709 >> 172.21.15.89:0/3745865709 conn(0x7fb540066b00 msgr2=0x7fb54006ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:01.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.491+0000 7fb54545c640 1 -- 172.21.15.89:0/3745865709 shutdown_connections 2024-03-29T11:35:01.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.491+0000 7fb54545c640 1 -- 172.21.15.89:0/3745865709 wait complete. 2024-03-29T11:35:01.491 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.491+0000 7fb54545c640 1 librados: shutdown 2024-03-29T11:35:01.500 DEBUG:teuthology.orchestra.run.smithi089:> sudo ceph --cluster ceph osd new fa9d435f-3f50-49a3-be51-8848398cdd8b 4 2024-03-29T11:35:01.531 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-29T11:35:01.533+0000 7f1f7fc2b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:35:01.549 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-29T11:35:01.551+0000 7fc853e3e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:35:01.605 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-29T11:35:01.607+0000 7f1f7fc2b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:35:01.618 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-29T11:35:01.621+0000 7fc853e3e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:35:01.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.632+0000 7f6ae7674640 10 monclient: get_monmap_and_config 2024-03-29T11:35:01.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.632+0000 7f6ae7674640 10 monclient: build_initial_monmap 2024-03-29T11:35:01.630 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.632+0000 7f6ae7674640 10 monclient: monmap: 2024-03-29T11:35:01.630 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 0 2024-03-29T11:35:01.630 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:01.630 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:35:01.633552+0000 2024-03-29T11:35:01.630 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:35:01.633552+0000 2024-03-29T11:35:01.630 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:01.630 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:01.630 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:35:01.631 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:35:01.631 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:35:01.631 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:01.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.632+0000 7f6ae7674640 1 Processor -- start 2024-03-29T11:35:01.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.632+0000 7f6ae7674640 1 -- start start 2024-03-29T11:35:01.639 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.632+0000 7f6ae7674640 10 monclient: init 2024-03-29T11:35:01.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.633+0000 7f6ae7674640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:01.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.633+0000 7f6ae7674640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:35:01.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.633+0000 7f6ae7674640 10 monclient(hunting): picked mon.noname-b con 0x7f6ae009ac40 addr v1:172.21.15.72:6790/0 2024-03-29T11:35:01.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.633+0000 7f6ae7674640 10 monclient(hunting): picked mon.noname-c con 0x7f6ae009c870 addr v1:172.21.15.89:6789/0 2024-03-29T11:35:01.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.633+0000 7f6ae7674640 10 monclient(hunting): picked mon.noname-a con 0x7f6ae00b4360 addr v1:172.21.15.72:6789/0 2024-03-29T11:35:01.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.633+0000 7f6ae7674640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ae009dec0 con 0x7f6ae00b4360 2024-03-29T11:35:01.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.633+0000 7f6ae7674640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ae009da30 con 0x7f6ae009c870 2024-03-29T11:35:01.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.633+0000 7f6ae7674640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ae00b39c0 con 0x7f6ae009ac40 2024-03-29T11:35:01.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.633+0000 7f6ae7674640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:01.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.633+0000 7f6ae7674640 10 monclient(hunting): authenticate will time out at 873.906602s 2024-03-29T11:35:01.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.633+0000 7f6ae4c0f640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f6ae009c870 0x7f6ae013f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57054/0 (socket says 172.21.15.89:57054) 2024-03-29T11:35:01.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.633+0000 7f6ae4c0f640 1 -- 172.21.15.89:0/608111949 learned_addr learned my addr 172.21.15.89:0/608111949 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-03-29T11:35:01.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.633+0000 7f6ad7fff640 1 -- 172.21.15.89:0/608111949 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 893507027 0 0) 0x7f6ae009da30 con 0x7f6ae009c870 2024-03-29T11:35:01.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.633+0000 7f6ad7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.640 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.633+0000 7f6ad7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.633+0000 7f6ad7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.633+0000 7f6ad7fff640 10 monclient(hunting): my global_id is 4138 2024-03-29T11:35:01.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.633+0000 7f6ad7fff640 1 -- 172.21.15.89:0/608111949 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6acc003520 con 0x7f6ae009c870 2024-03-29T11:35:01.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.633+0000 7f6ad7fff640 1 -- 172.21.15.89:0/608111949 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3400822015 0 0) 0x7f6ae00b39c0 con 0x7f6ae009ac40 2024-03-29T11:35:01.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.633+0000 7f6ad7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.634+0000 7f6ad7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.634+0000 7f6ad7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.634+0000 7f6ad7fff640 10 monclient(hunting): my global_id is 4127 2024-03-29T11:35:01.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.634+0000 7f6ad7fff640 1 -- 172.21.15.89:0/608111949 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ae009da30 con 0x7f6ae009ac40 2024-03-29T11:35:01.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.634+0000 7f6ad7fff640 1 -- 172.21.15.89:0/608111949 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 668113810 0 0) 0x7f6ad8001490 con 0x7f6ae00b4360 2024-03-29T11:35:01.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.634+0000 7f6ad7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.634+0000 7f6ad7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.634+0000 7f6ad7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.634+0000 7f6ad7fff640 10 monclient(hunting): my global_id is 4134 2024-03-29T11:35:01.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.634+0000 7f6ad7fff640 1 -- 172.21.15.89:0/608111949 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ae00b39c0 con 0x7f6ae00b4360 2024-03-29T11:35:01.641 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.634+0000 7f6ad7fff640 1 -- 172.21.15.89:0/608111949 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1699893790 0 0) 0x7f6acc003520 con 0x7f6ae009c870 2024-03-29T11:35:01.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.634+0000 7f6ad7fff640 1 -- 172.21.15.89:0/608111949 >> v1:172.21.15.72:6790/0 conn(0x7f6ae009ac40 legacy=0x7f6ae00b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.634+0000 7f6ad7fff640 1 -- 172.21.15.89:0/608111949 >> v1:172.21.15.72:6789/0 conn(0x7f6ae00b4360 legacy=0x7f6ae014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.634+0000 7f6ad7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:35:01.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.634+0000 7f6ad7fff640 1 monclient: found mon.noname-c 2024-03-29T11:35:01.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.634+0000 7f6ad7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:01.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.634+0000 7f6ad7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.89:6789/0 2024-03-29T11:35:01.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.634+0000 7f6ad7fff640 1 -- 172.21.15.89:0/608111949 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ae00b5160 con 0x7f6ae009c870 2024-03-29T11:35:01.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.634+0000 7f6ad7fff640 10 monclient: _finish_auth 0 2024-03-29T11:35:01.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.634+0000 7f6ad7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:35:01.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.634+0000 7f6ad7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:01.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.634+0000 7f6ad7fff640 1 -- 172.21.15.89:0/608111949 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f6adc001690 con 0x7f6ae009c870 2024-03-29T11:35:01.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.634+0000 7f6ae7674640 5 monclient: authenticate success, global_id 4138 2024-03-29T11:35:01.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.634+0000 7f6ae7674640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:01.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.634+0000 7f6ad7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.642 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.634+0000 7f6ad7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:01.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.634+0000 7f6ad7fff640 10 monclient: dump: 2024-03-29T11:35:01.643 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:01.643 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:01.643 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.643 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.643 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.643 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:01.643 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:01.643 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:01.643 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:01.643 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:01.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.634+0000 7f6ae7674640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:01.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.634+0000 7f6ad7fff640 1 -- 172.21.15.89:0/608111949 <== mon.1 v1:172.21.15.89:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6adc001870 con 0x7f6ae009c870 2024-03-29T11:35:01.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.634+0000 7f6ad7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:01.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.634+0000 7f6ad7fff640 1 -- 172.21.15.89:0/608111949 <== mon.1 v1:172.21.15.89:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f6adc001c80 con 0x7f6ae009c870 2024-03-29T11:35:01.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.634+0000 7f6ad7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.643 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.634+0000 7f6ad7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:01.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.634+0000 7f6ad7fff640 10 monclient: dump: 2024-03-29T11:35:01.644 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:01.644 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:01.644 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.644 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.644 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.644 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:01.644 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:01.644 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:01.644 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:01.644 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:01.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.634+0000 7f6ae7674640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:01.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.634+0000 7f6ae7674640 4 set_mon_vals no callback set 2024-03-29T11:35:01.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.635+0000 7f6ae7674640 10 monclient: shutdown 2024-03-29T11:35:01.644 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.635+0000 7f6ae7674640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:01.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.635+0000 7f6ae7674640 1 -- 172.21.15.89:0/608111949 >> v1:172.21.15.89:6789/0 conn(0x7f6ae009c870 legacy=0x7f6ae013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.635+0000 7f6ae7674640 1 -- 172.21.15.89:0/608111949 shutdown_connections 2024-03-29T11:35:01.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.635+0000 7f6ae7674640 1 -- 172.21.15.89:0/608111949 >> 172.21.15.89:0/608111949 conn(0x7f6ae0066b00 msgr2=0x7f6ae0066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:01.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.635+0000 7f6ae7674640 1 -- 172.21.15.89:0/608111949 shutdown_connections 2024-03-29T11:35:01.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.635+0000 7f6ae7674640 1 -- 172.21.15.89:0/608111949 wait complete. 2024-03-29T11:35:01.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.635+0000 7f6ae7674640 10 monclient: build_initial_monmap 2024-03-29T11:35:01.645 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.635+0000 7f6ae7674640 10 monclient: monmap: 2024-03-29T11:35:01.645 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 0 2024-03-29T11:35:01.645 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:01.645 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 0.000000 2024-03-29T11:35:01.645 INFO:teuthology.orchestra.run.smithi089.stderr:created 0.000000 2024-03-29T11:35:01.645 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:01.645 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:01.645 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:35:01.646 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:35:01.646 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:35:01.646 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:01.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.636+0000 7f6ae7674640 1 librados: starting msgr at 2024-03-29T11:35:01.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.636+0000 7f6ae7674640 1 librados: starting objecter 2024-03-29T11:35:01.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.636+0000 7f6ae7674640 1 Processor -- start 2024-03-29T11:35:01.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.636+0000 7f6ae7674640 1 -- start start 2024-03-29T11:35:01.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.636+0000 7f6ae7674640 1 librados: setting wanted keys 2024-03-29T11:35:01.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.636+0000 7f6ae7674640 1 librados: calling monclient init 2024-03-29T11:35:01.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.636+0000 7f6ae7674640 10 monclient: init 2024-03-29T11:35:01.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.636+0000 7f6ae7674640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:01.646 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.636+0000 7f6ae7674640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:35:01.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.636+0000 7f6ae7674640 10 monclient(hunting): picked mon.noname-b con 0x7f6ae009ac40 addr v1:172.21.15.89:6789/0 2024-03-29T11:35:01.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.636+0000 7f6ae7674640 10 monclient(hunting): picked mon.noname-c con 0x7f6ae0140660 addr v1:172.21.15.72:6790/0 2024-03-29T11:35:01.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.636+0000 7f6ae7674640 10 monclient(hunting): picked mon.noname-a con 0x7f6ae0140f80 addr v1:172.21.15.72:6789/0 2024-03-29T11:35:01.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.636+0000 7f6ae7674640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ae01423a0 con 0x7f6ae0140f80 2024-03-29T11:35:01.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.636+0000 7f6ae7674640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ae0140140 con 0x7f6ae009ac40 2024-03-29T11:35:01.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.636+0000 7f6ae7674640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ae00b8040 con 0x7f6ae0140660 2024-03-29T11:35:01.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.636+0000 7f6ae7674640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:01.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.636+0000 7f6ae7674640 10 monclient(hunting): authenticate will time out at 873.909815s 2024-03-29T11:35:01.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.636+0000 7f6ae5410640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f6ae009ac40 0x7f6ae01d7480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57062/0 (socket says 172.21.15.89:57062) 2024-03-29T11:35:01.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.636+0000 7f6ae5410640 1 -- 172.21.15.89:0/2860288077 learned_addr learned my addr 172.21.15.89:0/2860288077 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-03-29T11:35:01.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.636+0000 7f6ad5ffb640 10 client.?.objecter ms_handle_connect 0x7f6ae009ac40 2024-03-29T11:35:01.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.636+0000 7f6ad5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:01.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 10 client.?.objecter ms_handle_connect 0x7f6ae0140660 2024-03-29T11:35:01.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:01.647 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 1 -- 172.21.15.89:0/2860288077 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 22576909 0 0) 0x7f6ae0140140 con 0x7f6ae009ac40 2024-03-29T11:35:01.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 10 monclient(hunting): my global_id is 4141 2024-03-29T11:35:01.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 1 -- 172.21.15.89:0/2860288077 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ac4004eb0 con 0x7f6ae009ac40 2024-03-29T11:35:01.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 10 client.?.objecter ms_handle_connect 0x7f6ae0140f80 2024-03-29T11:35:01.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:01.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 1 -- 172.21.15.89:0/2860288077 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1616347645 0 0) 0x7f6ae00b8040 con 0x7f6ae0140660 2024-03-29T11:35:01.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 10 monclient(hunting): my global_id is 4130 2024-03-29T11:35:01.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 1 -- 172.21.15.89:0/2860288077 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ae0140140 con 0x7f6ae0140660 2024-03-29T11:35:01.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 1 -- 172.21.15.89:0/2860288077 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3111629242 0 0) 0x7f6ae01423a0 con 0x7f6ae0140f80 2024-03-29T11:35:01.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.648 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 10 monclient(hunting): my global_id is 4137 2024-03-29T11:35:01.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 1 -- 172.21.15.89:0/2860288077 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ae00b8040 con 0x7f6ae0140f80 2024-03-29T11:35:01.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 1 -- 172.21.15.89:0/2860288077 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3029807762 0 0) 0x7f6ac4004eb0 con 0x7f6ae009ac40 2024-03-29T11:35:01.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 1 -- 172.21.15.89:0/2860288077 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ae01423a0 con 0x7f6ae009ac40 2024-03-29T11:35:01.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 1 -- 172.21.15.89:0/2860288077 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f6ad00019b0 con 0x7f6ae009ac40 2024-03-29T11:35:01.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:01.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 10 monclient(hunting): dump: 2024-03-29T11:35:01.649 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:01.649 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:01.649 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.649 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.649 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.649 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:01.649 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:01.650 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:01.650 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:01.650 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:01.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:01.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 1 -- 172.21.15.89:0/2860288077 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4272233347 0 0) 0x7f6ae0140140 con 0x7f6ae0140660 2024-03-29T11:35:01.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 1 -- 172.21.15.89:0/2860288077 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ac4004eb0 con 0x7f6ae0140660 2024-03-29T11:35:01.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 1 -- 172.21.15.89:0/2860288077 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f6adc001b80 con 0x7f6ae0140660 2024-03-29T11:35:01.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:01.650 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 10 monclient(hunting): dump: 2024-03-29T11:35:01.650 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:01.650 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:01.650 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.650 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.650 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.650 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:01.651 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:01.651 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:01.651 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:01.651 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:01.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 1 -- 172.21.15.89:0/2860288077 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 847613425 0 0) 0x7f6ae00b8040 con 0x7f6ae0140f80 2024-03-29T11:35:01.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 1 -- 172.21.15.89:0/2860288077 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ae0140140 con 0x7f6ae0140f80 2024-03-29T11:35:01.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 1 -- 172.21.15.89:0/2860288077 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f6ad8001bc0 con 0x7f6ae0140f80 2024-03-29T11:35:01.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:01.651 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 10 monclient(hunting): dump: 2024-03-29T11:35:01.651 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:01.651 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:01.651 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.651 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.651 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.651 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:01.652 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:01.652 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:01.652 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:01.652 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:01.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 1 -- 172.21.15.89:0/2860288077 <== mon.1 v1:172.21.15.89:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4135840808 0 0) 0x7f6ae01423a0 con 0x7f6ae009ac40 2024-03-29T11:35:01.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 1 -- 172.21.15.89:0/2860288077 >> v1:172.21.15.72:6790/0 conn(0x7f6ae0140660 legacy=0x7f6ae00b1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 1 -- 172.21.15.89:0/2860288077 >> v1:172.21.15.72:6789/0 conn(0x7f6ae0140f80 legacy=0x7f6ae01e2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:35:01.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 1 monclient: found mon.b 2024-03-29T11:35:01.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:01.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:01.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 1 -- 172.21.15.89:0/2860288077 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ae00b81d0 con 0x7f6ae009ac40 2024-03-29T11:35:01.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:35:01.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:35:01.652 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:01.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ae7674640 5 monclient: authenticate success, global_id 4141 2024-03-29T11:35:01.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ae7674640 10 monclient: _renew_subs 2024-03-29T11:35:01.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ae7674640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:01.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ae7674640 1 -- 172.21.15.89:0/2860288077 --> v1:172.21.15.89:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ae01d7b90 con 0x7f6ae009ac40 2024-03-29T11:35:01.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 1 -- 172.21.15.89:0/2860288077 <== mon.1 v1:172.21.15.89:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ad0001c50 con 0x7f6ae009ac40 2024-03-29T11:35:01.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:01.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 1 -- 172.21.15.89:0/2860288077 <== mon.1 v1:172.21.15.89:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f6ad0003a60 con 0x7f6ae009ac40 2024-03-29T11:35:01.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:01.653 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 10 monclient: dump: 2024-03-29T11:35:01.653 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:01.653 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:01.653 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.653 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.653 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.654 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:01.654 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:01.654 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:01.654 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:01.654 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:01.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 1 -- 172.21.15.89:0/2860288077 <== mon.1 v1:172.21.15.89:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6ad0003fd0 con 0x7f6ae009ac40 2024-03-29T11:35:01.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ae7674640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:01.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad77fe640 4 set_mon_vals no callback set 2024-03-29T11:35:01.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ae7674640 10 monclient: _renew_subs 2024-03-29T11:35:01.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ae7674640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:01.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ae7674640 1 -- 172.21.15.89:0/2860288077 --> v1:172.21.15.89:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ae00b19f0 con 0x7f6ae009ac40 2024-03-29T11:35:01.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ae7674640 1 librados: init done 2024-03-29T11:35:01.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 1 -- 172.21.15.89:0/2860288077 <== mon.1 v1:172.21.15.89:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 2782839222 0 0) 0x7f6ad00057c0 con 0x7f6ae009ac40 2024-03-29T11:35:01.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 10 client.4141.objecter ms_dispatch 0x7f6ae0000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T11:35:01.654 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 3 client.4141.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T11:35:01.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.637+0000 7f6ad5ffb640 3 client.4141.objecter handle_osd_map decoding full epoch 6 2024-03-29T11:35:01.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.638+0000 7f6ad5ffb640 20 client.4141.objecter dump_active .. 0 homeless 2024-03-29T11:35:01.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.638+0000 7f6ae7674640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:01.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.638+0000 7f6ae7674640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:01.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.638+0000 7f6ae7674640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:01.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.638+0000 7f6ae7674640 1 -- 172.21.15.89:0/2860288077 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ae013ffa0 con 0x7f6ae009ac40 2024-03-29T11:35:01.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.640+0000 7f6ad5ffb640 1 -- 172.21.15.89:0/2860288077 <== mon.1 v1:172.21.15.89:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6ae013ffa0 con 0x7f6ae009ac40 2024-03-29T11:35:01.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.640+0000 7f6ad5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:01.655 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.640+0000 7f6ad5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:01.655 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-29T11:35:01.656+0000 7f1f7fc2b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:35:01.664 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-29T11:35:01.667+0000 7fc853e3e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:35:01.706 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-29T11:35:01.708+0000 7f1f7fc2b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:35:01.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.711+0000 7f6ae7674640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "fa9d435f-3f50-49a3-be51-8848398cdd8b", "id": 4}] 2024-03-29T11:35:01.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.711+0000 7f6ae7674640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "fa9d435f-3f50-49a3-be51-8848398cdd8b", "id": 4}] 2024-03-29T11:35:01.709 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.711+0000 7f6ae7674640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:01.710 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.711+0000 7f6ae7674640 1 -- 172.21.15.89:0/2860288077 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "osd new", "uuid": "fa9d435f-3f50-49a3-be51-8848398cdd8b", "id": 4} v 0) v1 -- 0x7f6ae01402d0 con 0x7f6ae009ac40 2024-03-29T11:35:01.717 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-29T11:35:01.719+0000 7fc853e3e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:35:01.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.718+0000 7f6ad5ffb640 1 -- 172.21.15.89:0/2860288077 <== mon.1 v1:172.21.15.89:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fa9d435f-3f50-49a3-be51-8848398cdd8b", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 1474828312 0 1697012583) 0x7f6ad0005af0 con 0x7f6ae009ac40 2024-03-29T11:35:01.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.719+0000 7f6ad5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "fa9d435f-3f50-49a3-be51-8848398cdd8b", "id": 4}] 2024-03-29T11:35:01.717 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.719+0000 7f6ad5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:01.717 INFO:teuthology.orchestra.run.smithi089.stdout:4 2024-03-29T11:35:01.718 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.719+0000 7f6ae7674640 10 librados: watch_flush enter 2024-03-29T11:35:01.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.719+0000 7f6ae7674640 10 librados: watch_flush exit 2024-03-29T11:35:01.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.719+0000 7f6ae7674640 20 client.4141.objecter shutdown clearing up homeless session... 2024-03-29T11:35:01.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.719+0000 7f6ae7674640 10 client.4141.objecter successfully canceled tick 2024-03-29T11:35:01.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.720+0000 7f6ae7674640 10 monclient: shutdown 2024-03-29T11:35:01.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.720+0000 7f6ae7674640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:01.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.720+0000 7f6ae7674640 1 -- 172.21.15.89:0/2860288077 >> v1:172.21.15.89:6789/0 conn(0x7f6ae009ac40 legacy=0x7f6ae01d7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.720+0000 7f6ae7674640 1 -- 172.21.15.89:0/2860288077 shutdown_connections 2024-03-29T11:35:01.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.720+0000 7f6ae7674640 1 -- 172.21.15.89:0/2860288077 >> 172.21.15.89:0/2860288077 conn(0x7f6ae0066b00 msgr2=0x7f6ae013c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:01.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.720+0000 7f6ae7674640 1 -- 172.21.15.89:0/2860288077 shutdown_connections 2024-03-29T11:35:01.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.720+0000 7f6ae7674640 1 -- 172.21.15.89:0/2860288077 wait complete. 2024-03-29T11:35:01.719 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.720+0000 7f6ae7674640 1 librados: shutdown 2024-03-29T11:35:01.728 DEBUG:teuthology.orchestra.run.smithi089:> sudo ceph --cluster ceph osd new e044742d-5adf-4444-9725-f7a9ac404cf9 5 2024-03-29T11:35:01.758 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-29T11:35:01.760+0000 7f1f7fc2b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:35:01.763 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-29T11:35:01.766+0000 7fc853e3e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:35:01.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.830+0000 7f4d1a0c8640 10 monclient: get_monmap_and_config 2024-03-29T11:35:01.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.830+0000 7f4d1a0c8640 10 monclient: build_initial_monmap 2024-03-29T11:35:01.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.830+0000 7f4d1a0c8640 10 monclient: monmap: 2024-03-29T11:35:01.829 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 0 2024-03-29T11:35:01.829 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:01.829 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:35:01.832042+0000 2024-03-29T11:35:01.829 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:35:01.832042+0000 2024-03-29T11:35:01.829 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:01.829 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:01.829 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:35:01.829 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:35:01.829 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:35:01.829 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:01.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.831+0000 7f4d1a0c8640 1 Processor -- start 2024-03-29T11:35:01.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.831+0000 7f4d1a0c8640 1 -- start start 2024-03-29T11:35:01.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.831+0000 7f4d1a0c8640 10 monclient: init 2024-03-29T11:35:01.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.831+0000 7f4d1a0c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:01.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.831+0000 7f4d1a0c8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:35:01.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.831+0000 7f4d1a0c8640 10 monclient(hunting): picked mon.noname-c con 0x7f4d1409ac40 addr v1:172.21.15.89:6789/0 2024-03-29T11:35:01.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.831+0000 7f4d1a0c8640 10 monclient(hunting): picked mon.noname-a con 0x7f4d1409c870 addr v1:172.21.15.72:6789/0 2024-03-29T11:35:01.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.831+0000 7f4d1a0c8640 10 monclient(hunting): picked mon.noname-b con 0x7f4d140b4360 addr v1:172.21.15.72:6790/0 2024-03-29T11:35:01.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.831+0000 7f4d1a0c8640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d1409dec0 con 0x7f4d1409c870 2024-03-29T11:35:01.838 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d1a0c8640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d1409da30 con 0x7f4d1409ac40 2024-03-29T11:35:01.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d1a0c8640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d140b39c0 con 0x7f4d140b4360 2024-03-29T11:35:01.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d1a0c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:01.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d1a0c8640 10 monclient(hunting): authenticate will time out at 874.105021s 2024-03-29T11:35:01.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d137fe640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f4d1409ac40 0x7f4d140b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57068/0 (socket says 172.21.15.89:57068) 2024-03-29T11:35:01.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d137fe640 1 -- 172.21.15.89:0/709621960 learned_addr learned my addr 172.21.15.89:0/709621960 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-03-29T11:35:01.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d127fc640 1 -- 172.21.15.89:0/709621960 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 597292098 0 0) 0x7f4d1409da30 con 0x7f4d1409ac40 2024-03-29T11:35:01.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d127fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d127fc640 10 monclient(hunting): my global_id is 4144 2024-03-29T11:35:01.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d127fc640 1 -- 172.21.15.89:0/709621960 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cfc003520 con 0x7f4d1409ac40 2024-03-29T11:35:01.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d127fc640 1 -- 172.21.15.89:0/709621960 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2306065085 0 0) 0x7f4d04001110 con 0x7f4d1409c870 2024-03-29T11:35:01.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d127fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.839 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d127fc640 10 monclient(hunting): my global_id is 4140 2024-03-29T11:35:01.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d127fc640 1 -- 172.21.15.89:0/709621960 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d1409da30 con 0x7f4d1409c870 2024-03-29T11:35:01.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d127fc640 1 -- 172.21.15.89:0/709621960 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2128391075 0 0) 0x7f4d140b39c0 con 0x7f4d140b4360 2024-03-29T11:35:01.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d127fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d127fc640 10 monclient(hunting): my global_id is 4133 2024-03-29T11:35:01.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d127fc640 1 -- 172.21.15.89:0/709621960 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d04001110 con 0x7f4d140b4360 2024-03-29T11:35:01.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d127fc640 1 -- 172.21.15.89:0/709621960 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1254587729 0 0) 0x7f4cfc003520 con 0x7f4d1409ac40 2024-03-29T11:35:01.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d127fc640 1 -- 172.21.15.89:0/709621960 >> v1:172.21.15.72:6790/0 conn(0x7f4d140b4360 legacy=0x7f4d1414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d127fc640 1 -- 172.21.15.89:0/709621960 >> v1:172.21.15.72:6789/0 conn(0x7f4d1409c870 legacy=0x7f4d1413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d127fc640 10 monclient: _finish_hunting 0 2024-03-29T11:35:01.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d127fc640 1 monclient: found mon.noname-c 2024-03-29T11:35:01.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:01.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d127fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.89:6789/0 2024-03-29T11:35:01.840 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d127fc640 1 -- 172.21.15.89:0/709621960 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d140b5160 con 0x7f4d1409ac40 2024-03-29T11:35:01.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d127fc640 10 monclient: _finish_auth 0 2024-03-29T11:35:01.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d127fc640 10 monclient: _check_auth_tickets 2024-03-29T11:35:01.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:01.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d127fc640 1 -- 172.21.15.89:0/709621960 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f4d08001690 con 0x7f4d1409ac40 2024-03-29T11:35:01.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d1a0c8640 5 monclient: authenticate success, global_id 4144 2024-03-29T11:35:01.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d1a0c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:01.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d127fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:01.841 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d127fc640 10 monclient: dump: 2024-03-29T11:35:01.841 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:01.841 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:01.841 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.841 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.841 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.841 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:01.841 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:01.842 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:01.842 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:01.842 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:01.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d1a0c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:01.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d127fc640 1 -- 172.21.15.89:0/709621960 <== mon.1 v1:172.21.15.89:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d08001870 con 0x7f4d1409ac40 2024-03-29T11:35:01.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:01.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d127fc640 1 -- 172.21.15.89:0/709621960 <== mon.1 v1:172.21.15.89:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f4d08001c80 con 0x7f4d1409ac40 2024-03-29T11:35:01.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d1a0c8640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:01.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d1a0c8640 4 set_mon_vals no callback set 2024-03-29T11:35:01.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d127fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:01.842 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.832+0000 7f4d127fc640 10 monclient: dump: 2024-03-29T11:35:01.842 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:01.842 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:01.842 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.843 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.843 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.843 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:01.843 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:01.843 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:01.843 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:01.843 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:01.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.833+0000 7f4d1a0c8640 10 monclient: shutdown 2024-03-29T11:35:01.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.833+0000 7f4d1a0c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:01.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.833+0000 7f4d1a0c8640 1 -- 172.21.15.89:0/709621960 >> v1:172.21.15.89:6789/0 conn(0x7f4d1409ac40 legacy=0x7f4d140b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.833+0000 7f4d1a0c8640 1 -- 172.21.15.89:0/709621960 shutdown_connections 2024-03-29T11:35:01.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.833+0000 7f4d1a0c8640 1 -- 172.21.15.89:0/709621960 >> 172.21.15.89:0/709621960 conn(0x7f4d14066b00 msgr2=0x7f4d14066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:01.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.833+0000 7f4d1a0c8640 1 -- 172.21.15.89:0/709621960 shutdown_connections 2024-03-29T11:35:01.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.833+0000 7f4d1a0c8640 1 -- 172.21.15.89:0/709621960 wait complete. 2024-03-29T11:35:01.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.833+0000 7f4d1a0c8640 10 monclient: build_initial_monmap 2024-03-29T11:35:01.843 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.833+0000 7f4d1a0c8640 10 monclient: monmap: 2024-03-29T11:35:01.844 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 0 2024-03-29T11:35:01.844 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:01.844 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 0.000000 2024-03-29T11:35:01.844 INFO:teuthology.orchestra.run.smithi089.stderr:created 0.000000 2024-03-29T11:35:01.844 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:01.844 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:01.844 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:35:01.844 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:35:01.844 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:35:01.844 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:01.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.834+0000 7f4d1a0c8640 1 librados: starting msgr at 2024-03-29T11:35:01.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.834+0000 7f4d1a0c8640 1 librados: starting objecter 2024-03-29T11:35:01.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.834+0000 7f4d1a0c8640 1 Processor -- start 2024-03-29T11:35:01.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.834+0000 7f4d1a0c8640 1 -- start start 2024-03-29T11:35:01.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.834+0000 7f4d1a0c8640 1 librados: setting wanted keys 2024-03-29T11:35:01.844 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.834+0000 7f4d1a0c8640 1 librados: calling monclient init 2024-03-29T11:35:01.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.834+0000 7f4d1a0c8640 10 monclient: init 2024-03-29T11:35:01.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.834+0000 7f4d1a0c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:01.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.834+0000 7f4d1a0c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:35:01.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.834+0000 7f4d1a0c8640 10 monclient(hunting): picked mon.noname-c con 0x7f4d1409ac40 addr v1:172.21.15.72:6790/0 2024-03-29T11:35:01.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.834+0000 7f4d1a0c8640 10 monclient(hunting): picked mon.noname-b con 0x7f4d141ccb70 addr v1:172.21.15.89:6789/0 2024-03-29T11:35:01.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.834+0000 7f4d1a0c8640 10 monclient(hunting): picked mon.noname-a con 0x7f4d141425f0 addr v1:172.21.15.72:6789/0 2024-03-29T11:35:01.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.834+0000 7f4d1a0c8640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d141cd520 con 0x7f4d141425f0 2024-03-29T11:35:01.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.834+0000 7f4d1a0c8640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d140acd00 con 0x7f4d141ccb70 2024-03-29T11:35:01.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.834+0000 7f4d1a0c8640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d141cd000 con 0x7f4d1409ac40 2024-03-29T11:35:01.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.834+0000 7f4d1a0c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:01.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.834+0000 7f4d1a0c8640 10 monclient(hunting): authenticate will time out at 874.107936s 2024-03-29T11:35:01.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4d12ffd640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f4d141ccb70 0x7f4d1413adc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57080/0 (socket says 172.21.15.89:57080) 2024-03-29T11:35:01.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4d12ffd640 1 -- 172.21.15.89:0/1980029500 learned_addr learned my addr 172.21.15.89:0/1980029500 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-03-29T11:35:01.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 10 client.?.objecter ms_handle_connect 0x7f4d141ccb70 2024-03-29T11:35:01.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:01.845 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 10 client.?.objecter ms_handle_connect 0x7f4d1409ac40 2024-03-29T11:35:01.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:01.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 10 client.?.objecter ms_handle_connect 0x7f4d141425f0 2024-03-29T11:35:01.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:01.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 1 -- 172.21.15.89:0/1980029500 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4042579600 0 0) 0x7f4d140acd00 con 0x7f4d141ccb70 2024-03-29T11:35:01.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 10 monclient(hunting): my global_id is 4147 2024-03-29T11:35:01.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 1 -- 172.21.15.89:0/1980029500 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d00004eb0 con 0x7f4d141ccb70 2024-03-29T11:35:01.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 1 -- 172.21.15.89:0/1980029500 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4016474667 0 0) 0x7f4d141cd000 con 0x7f4d1409ac40 2024-03-29T11:35:01.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 10 monclient(hunting): my global_id is 4136 2024-03-29T11:35:01.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 1 -- 172.21.15.89:0/1980029500 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d140acd00 con 0x7f4d1409ac40 2024-03-29T11:35:01.846 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 1 -- 172.21.15.89:0/1980029500 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2481816359 0 0) 0x7f4d141cd520 con 0x7f4d141425f0 2024-03-29T11:35:01.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:01.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:01.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:01.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 10 monclient(hunting): my global_id is 4143 2024-03-29T11:35:01.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 1 -- 172.21.15.89:0/1980029500 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d141cd000 con 0x7f4d141425f0 2024-03-29T11:35:01.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 1 -- 172.21.15.89:0/1980029500 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2741846286 0 0) 0x7f4d00004eb0 con 0x7f4d141ccb70 2024-03-29T11:35:01.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 1 -- 172.21.15.89:0/1980029500 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d141cd520 con 0x7f4d141ccb70 2024-03-29T11:35:01.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 1 -- 172.21.15.89:0/1980029500 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1115457481 0 0) 0x7f4d140acd00 con 0x7f4d1409ac40 2024-03-29T11:35:01.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 1 -- 172.21.15.89:0/1980029500 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d00004eb0 con 0x7f4d1409ac40 2024-03-29T11:35:01.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 1 -- 172.21.15.89:0/1980029500 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f4d040019a0 con 0x7f4d141ccb70 2024-03-29T11:35:01.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:01.847 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 10 monclient(hunting): dump: 2024-03-29T11:35:01.847 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:01.847 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:01.847 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.848 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.848 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.848 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:01.848 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:01.848 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:01.848 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:01.848 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:01.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:01.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 1 -- 172.21.15.89:0/1980029500 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f4d08001b80 con 0x7f4d1409ac40 2024-03-29T11:35:01.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:01.848 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 10 monclient(hunting): dump: 2024-03-29T11:35:01.848 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:01.848 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:01.848 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.848 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.849 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.849 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:01.849 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:01.849 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:01.849 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:01.849 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:01.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 1 -- 172.21.15.89:0/1980029500 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1348073983 0 0) 0x7f4d141cd000 con 0x7f4d141425f0 2024-03-29T11:35:01.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 1 -- 172.21.15.89:0/1980029500 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d140acd00 con 0x7f4d141425f0 2024-03-29T11:35:01.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 1 -- 172.21.15.89:0/1980029500 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f4d0c001b50 con 0x7f4d141425f0 2024-03-29T11:35:01.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:01.849 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 10 monclient(hunting): dump: 2024-03-29T11:35:01.849 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:01.849 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:01.849 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.850 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.850 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.850 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:01.850 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:01.850 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:01.850 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:01.850 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:01.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 1 -- 172.21.15.89:0/1980029500 <== mon.1 v1:172.21.15.89:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2334440000 0 0) 0x7f4d141cd520 con 0x7f4d141ccb70 2024-03-29T11:35:01.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 1 -- 172.21.15.89:0/1980029500 >> v1:172.21.15.72:6790/0 conn(0x7f4d1409ac40 legacy=0x7f4d141da690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 1 -- 172.21.15.89:0/1980029500 >> v1:172.21.15.72:6789/0 conn(0x7f4d141425f0 legacy=0x7f4d141e29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:35:01.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 1 monclient: found mon.b 2024-03-29T11:35:01.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:01.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:01.850 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 1 -- 172.21.15.89:0/1980029500 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d140b1af0 con 0x7f4d141ccb70 2024-03-29T11:35:01.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 10 monclient: _finish_auth 0 2024-03-29T11:35:01.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:35:01.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:01.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4d1a0c8640 5 monclient: authenticate success, global_id 4147 2024-03-29T11:35:01.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4d1a0c8640 10 monclient: _renew_subs 2024-03-29T11:35:01.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4d1a0c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:01.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4d1a0c8640 1 -- 172.21.15.89:0/1980029500 --> v1:172.21.15.89:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d141cc300 con 0x7f4d141ccb70 2024-03-29T11:35:01.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4d1a0c8640 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:01.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4d1a0c8640 10 monclient: _renew_subs 2024-03-29T11:35:01.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4d1a0c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:01.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4d1a0c8640 1 -- 172.21.15.89:0/1980029500 --> v1:172.21.15.89:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d141daf80 con 0x7f4d141ccb70 2024-03-29T11:35:01.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4d1a0c8640 1 librados: init done 2024-03-29T11:35:01.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 1 -- 172.21.15.89:0/1980029500 <== mon.1 v1:172.21.15.89:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d04001c80 con 0x7f4d141ccb70 2024-03-29T11:35:01.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:01.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 1 -- 172.21.15.89:0/1980029500 <== mon.1 v1:172.21.15.89:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f4d04001b00 con 0x7f4d141ccb70 2024-03-29T11:35:01.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:01.851 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:01.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 10 monclient: dump: 2024-03-29T11:35:01.852 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:01.852 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:01.852 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.852 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:01.852 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:01.852 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:01.852 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:01.852 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:01.852 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:01.852 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:01.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4d11ffb640 4 set_mon_vals no callback set 2024-03-29T11:35:01.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 1 -- 172.21.15.89:0/1980029500 <== mon.1 v1:172.21.15.89:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f4d04003eb0 con 0x7f4d141ccb70 2024-03-29T11:35:01.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 1 -- 172.21.15.89:0/1980029500 <== mon.1 v1:172.21.15.89:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 1959669064 0 0) 0x7f4d040067d0 con 0x7f4d141ccb70 2024-03-29T11:35:01.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 10 client.4147.objecter ms_dispatch 0x7f4d14000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T11:35:01.852 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 3 client.4147.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T11:35:01.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 3 client.4147.objecter handle_osd_map decoding full epoch 7 2024-03-29T11:35:01.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.835+0000 7f4cfbfff640 20 client.4147.objecter dump_active .. 0 homeless 2024-03-29T11:35:01.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.836+0000 7f4d1a0c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:01.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.836+0000 7f4d1a0c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:01.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.836+0000 7f4d1a0c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:01.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.836+0000 7f4d1a0c8640 1 -- 172.21.15.89:0/1980029500 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cf4005080 con 0x7f4d141ccb70 2024-03-29T11:35:01.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.838+0000 7f4cfbfff640 1 -- 172.21.15.89:0/1980029500 <== mon.1 v1:172.21.15.89:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f4d040042f0 con 0x7f4d141ccb70 2024-03-29T11:35:01.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.838+0000 7f4cfbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:01.853 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.838+0000 7f4cfbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:01.908 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.910+0000 7f4d1a0c8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e044742d-5adf-4444-9725-f7a9ac404cf9", "id": 5}] 2024-03-29T11:35:01.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.910+0000 7f4d1a0c8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e044742d-5adf-4444-9725-f7a9ac404cf9", "id": 5}] 2024-03-29T11:35:01.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.910+0000 7f4d1a0c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:01.909 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.910+0000 7f4d1a0c8640 1 -- 172.21.15.89:0/1980029500 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "osd new", "uuid": "e044742d-5adf-4444-9725-f7a9ac404cf9", "id": 5} v 0) v1 -- 0x7f4cf40058b0 con 0x7f4d141ccb70 2024-03-29T11:35:01.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.916+0000 7f4cfbfff640 1 -- 172.21.15.89:0/1980029500 <== mon.1 v1:172.21.15.89:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e044742d-5adf-4444-9725-f7a9ac404cf9", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 1629354240 0 1988412176) 0x7f4d04006320 con 0x7f4d141ccb70 2024-03-29T11:35:01.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.916+0000 7f4cfbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e044742d-5adf-4444-9725-f7a9ac404cf9", "id": 5}] 2024-03-29T11:35:01.914 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.916+0000 7f4cfbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:01.915 INFO:teuthology.orchestra.run.smithi089.stdout:5 2024-03-29T11:35:01.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.917+0000 7f4d1a0c8640 10 librados: watch_flush enter 2024-03-29T11:35:01.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.917+0000 7f4d1a0c8640 10 librados: watch_flush exit 2024-03-29T11:35:01.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.917+0000 7f4d1a0c8640 20 client.4147.objecter shutdown clearing up homeless session... 2024-03-29T11:35:01.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.917+0000 7f4d1a0c8640 10 client.4147.objecter successfully canceled tick 2024-03-29T11:35:01.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.917+0000 7f4d1a0c8640 10 monclient: shutdown 2024-03-29T11:35:01.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.917+0000 7f4d1a0c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:01.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.917+0000 7f4d1a0c8640 1 -- 172.21.15.89:0/1980029500 >> v1:172.21.15.89:6789/0 conn(0x7f4d141ccb70 legacy=0x7f4d1413adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:01.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.917+0000 7f4d1a0c8640 1 -- 172.21.15.89:0/1980029500 shutdown_connections 2024-03-29T11:35:01.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.917+0000 7f4d1a0c8640 1 -- 172.21.15.89:0/1980029500 >> 172.21.15.89:0/1980029500 conn(0x7f4d14066b00 msgr2=0x7f4d14066ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:01.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.917+0000 7f4d1a0c8640 1 -- 172.21.15.89:0/1980029500 shutdown_connections 2024-03-29T11:35:01.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.917+0000 7f4d1a0c8640 1 -- 172.21.15.89:0/1980029500 wait complete. 2024-03-29T11:35:01.916 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:01.917+0000 7f4d1a0c8640 1 librados: shutdown 2024-03-29T11:35:01.925 DEBUG:teuthology.orchestra.run.smithi089:> sudo ceph --cluster ceph osd new 7c9b4262-0624-4322-b4be-78b105d0608e 6 2024-03-29T11:35:02.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.070+0000 7f001d8cf640 10 monclient: get_monmap_and_config 2024-03-29T11:35:02.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.070+0000 7f001d8cf640 10 monclient: build_initial_monmap 2024-03-29T11:35:02.068 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.070+0000 7f001d8cf640 10 monclient: monmap: 2024-03-29T11:35:02.068 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 0 2024-03-29T11:35:02.068 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:02.068 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:35:02.071716+0000 2024-03-29T11:35:02.068 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:35:02.071716+0000 2024-03-29T11:35:02.069 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:02.069 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:02.069 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:35:02.069 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:35:02.069 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:35:02.069 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:02.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.070+0000 7f001d8cf640 1 Processor -- start 2024-03-29T11:35:02.077 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.070+0000 7f001d8cf640 1 -- start start 2024-03-29T11:35:02.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.070+0000 7f001d8cf640 10 monclient: init 2024-03-29T11:35:02.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.071+0000 7f001d8cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:02.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.071+0000 7f001d8cf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:35:02.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.071+0000 7f001d8cf640 10 monclient(hunting): picked mon.noname-c con 0x7f00180b8240 addr v1:172.21.15.89:6789/0 2024-03-29T11:35:02.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.071+0000 7f001d8cf640 10 monclient(hunting): picked mon.noname-a con 0x7f00180b1f60 addr v1:172.21.15.72:6789/0 2024-03-29T11:35:02.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.071+0000 7f001d8cf640 10 monclient(hunting): picked mon.noname-b con 0x7f001809c8c0 addr v1:172.21.15.72:6790/0 2024-03-29T11:35:02.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.071+0000 7f001d8cf640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f001809bd80 con 0x7f00180b1f60 2024-03-29T11:35:02.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.071+0000 7f001d8cf640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f001809b960 con 0x7f00180b8240 2024-03-29T11:35:02.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.071+0000 7f001d8cf640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00180b2ec0 con 0x7f001809c8c0 2024-03-29T11:35:02.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.071+0000 7f001d8cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:02.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.071+0000 7f001d8cf640 10 monclient(hunting): authenticate will time out at 874.344553s 2024-03-29T11:35:02.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.071+0000 7f0016ffd640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f00180b8240 0x7f001813f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57090/0 (socket says 172.21.15.89:57090) 2024-03-29T11:35:02.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.071+0000 7f0016ffd640 1 -- 172.21.15.89:0/1934115897 learned_addr learned my addr 172.21.15.89:0/1934115897 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-03-29T11:35:02.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.071+0000 7f0015ffb640 1 -- 172.21.15.89:0/1934115897 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1531994366 0 0) 0x7f001809b960 con 0x7f00180b8240 2024-03-29T11:35:02.078 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.071+0000 7f0015ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:02.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.071+0000 7f0015ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:02.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.071+0000 7f0015ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:02.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.071+0000 7f0015ffb640 10 monclient(hunting): my global_id is 4150 2024-03-29T11:35:02.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.071+0000 7f0015ffb640 1 -- 172.21.15.89:0/1934115897 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0004003520 con 0x7f00180b8240 2024-03-29T11:35:02.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f0015ffb640 1 -- 172.21.15.89:0/1934115897 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1993231772 0 0) 0x7f00180b2ec0 con 0x7f001809c8c0 2024-03-29T11:35:02.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f0015ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:02.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f0015ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:02.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f0015ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:02.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f0015ffb640 10 monclient(hunting): my global_id is 4139 2024-03-29T11:35:02.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f0015ffb640 1 -- 172.21.15.89:0/1934115897 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f001809b960 con 0x7f001809c8c0 2024-03-29T11:35:02.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f0015ffb640 1 -- 172.21.15.89:0/1934115897 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 505216154 0 0) 0x7f001809bd80 con 0x7f00180b1f60 2024-03-29T11:35:02.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f0015ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:02.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f0015ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:02.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f0015ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:02.079 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f0015ffb640 10 monclient(hunting): my global_id is 4146 2024-03-29T11:35:02.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f0015ffb640 1 -- 172.21.15.89:0/1934115897 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00180b2ec0 con 0x7f00180b1f60 2024-03-29T11:35:02.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f0015ffb640 1 -- 172.21.15.89:0/1934115897 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2125578133 0 0) 0x7f0004003520 con 0x7f00180b8240 2024-03-29T11:35:02.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f0015ffb640 1 -- 172.21.15.89:0/1934115897 >> v1:172.21.15.72:6790/0 conn(0x7f001809c8c0 legacy=0x7f001814d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:02.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f0015ffb640 1 -- 172.21.15.89:0/1934115897 >> v1:172.21.15.72:6789/0 conn(0x7f00180b1f60 legacy=0x7f001809c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:02.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f0015ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:35:02.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f0015ffb640 1 monclient: found mon.noname-c 2024-03-29T11:35:02.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f0015ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:02.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f0015ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.89:6789/0 2024-03-29T11:35:02.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f0015ffb640 1 -- 172.21.15.89:0/1934115897 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f001813fb60 con 0x7f00180b8240 2024-03-29T11:35:02.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f0015ffb640 10 monclient: _finish_auth 0 2024-03-29T11:35:02.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f0015ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:35:02.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f0015ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:02.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f001d8cf640 5 monclient: authenticate success, global_id 4150 2024-03-29T11:35:02.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f0015ffb640 1 -- 172.21.15.89:0/1934115897 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f0000001690 con 0x7f00180b8240 2024-03-29T11:35:02.080 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f001d8cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:02.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f0015ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:02.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f0015ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:02.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f0015ffb640 10 monclient: dump: 2024-03-29T11:35:02.081 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:02.081 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:02.081 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:02.081 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:02.081 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:02.081 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:02.081 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:02.081 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:02.081 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:02.081 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:02.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f001d8cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:02.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f0015ffb640 1 -- 172.21.15.89:0/1934115897 <== mon.1 v1:172.21.15.89:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0000001870 con 0x7f00180b8240 2024-03-29T11:35:02.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f0015ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:02.081 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f0015ffb640 1 -- 172.21.15.89:0/1934115897 <== mon.1 v1:172.21.15.89:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f0000001c80 con 0x7f00180b8240 2024-03-29T11:35:02.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f001d8cf640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:02.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f001d8cf640 4 set_mon_vals no callback set 2024-03-29T11:35:02.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f0015ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:02.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f0015ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:02.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f0015ffb640 10 monclient: dump: 2024-03-29T11:35:02.082 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:02.082 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:02.082 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:02.082 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:02.082 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:02.082 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:02.082 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:02.082 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:02.082 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:02.082 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:02.082 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f001d8cf640 10 monclient: shutdown 2024-03-29T11:35:02.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f001d8cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:02.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f001d8cf640 1 -- 172.21.15.89:0/1934115897 >> v1:172.21.15.89:6789/0 conn(0x7f00180b8240 legacy=0x7f001813f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:02.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f001d8cf640 1 -- 172.21.15.89:0/1934115897 shutdown_connections 2024-03-29T11:35:02.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f001d8cf640 1 -- 172.21.15.89:0/1934115897 >> 172.21.15.89:0/1934115897 conn(0x7f0018066b00 msgr2=0x7f0018066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:02.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f001d8cf640 1 -- 172.21.15.89:0/1934115897 shutdown_connections 2024-03-29T11:35:02.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.072+0000 7f001d8cf640 1 -- 172.21.15.89:0/1934115897 wait complete. 2024-03-29T11:35:02.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.073+0000 7f001d8cf640 10 monclient: build_initial_monmap 2024-03-29T11:35:02.083 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.073+0000 7f001d8cf640 10 monclient: monmap: 2024-03-29T11:35:02.083 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 0 2024-03-29T11:35:02.083 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:02.083 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 0.000000 2024-03-29T11:35:02.083 INFO:teuthology.orchestra.run.smithi089.stderr:created 0.000000 2024-03-29T11:35:02.083 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:02.083 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:02.084 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:35:02.084 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:35:02.084 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:35:02.084 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:02.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.073+0000 7f001d8cf640 1 librados: starting msgr at 2024-03-29T11:35:02.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.073+0000 7f001d8cf640 1 librados: starting objecter 2024-03-29T11:35:02.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.073+0000 7f001d8cf640 1 Processor -- start 2024-03-29T11:35:02.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.073+0000 7f001d8cf640 1 -- start start 2024-03-29T11:35:02.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.073+0000 7f001d8cf640 1 librados: setting wanted keys 2024-03-29T11:35:02.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.073+0000 7f001d8cf640 1 librados: calling monclient init 2024-03-29T11:35:02.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.073+0000 7f001d8cf640 10 monclient: init 2024-03-29T11:35:02.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.074+0000 7f001d8cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:02.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.074+0000 7f001d8cf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:35:02.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.074+0000 7f001d8cf640 10 monclient(hunting): picked mon.noname-a con 0x7f001809c8c0 addr v1:172.21.15.72:6789/0 2024-03-29T11:35:02.084 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.074+0000 7f001d8cf640 10 monclient(hunting): picked mon.noname-b con 0x7f00180b8240 addr v1:172.21.15.89:6789/0 2024-03-29T11:35:02.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.074+0000 7f001d8cf640 10 monclient(hunting): picked mon.noname-c con 0x7f0018146090 addr v1:172.21.15.72:6790/0 2024-03-29T11:35:02.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.074+0000 7f001d8cf640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0018131780 con 0x7f001809c8c0 2024-03-29T11:35:02.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.074+0000 7f001d8cf640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00181d4e60 con 0x7f00180b8240 2024-03-29T11:35:02.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.074+0000 7f001d8cf640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00181d8040 con 0x7f0018146090 2024-03-29T11:35:02.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.074+0000 7f001d8cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:02.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.074+0000 7f001d8cf640 10 monclient(hunting): authenticate will time out at 874.347549s 2024-03-29T11:35:02.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.074+0000 7f00167fc640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f00180b8240 0x7f00181d7930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57092/0 (socket says 172.21.15.89:57092) 2024-03-29T11:35:02.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.074+0000 7f00167fc640 1 -- 172.21.15.89:0/2059796483 learned_addr learned my addr 172.21.15.89:0/2059796483 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-03-29T11:35:02.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.074+0000 7f000ffff640 10 client.?.objecter ms_handle_connect 0x7f00180b8240 2024-03-29T11:35:02.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.074+0000 7f000ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:02.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.074+0000 7f000ffff640 10 client.?.objecter ms_handle_connect 0x7f001809c8c0 2024-03-29T11:35:02.085 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.074+0000 7f000ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:02.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.074+0000 7f000ffff640 10 client.?.objecter ms_handle_connect 0x7f0018146090 2024-03-29T11:35:02.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.074+0000 7f000ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:02.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.074+0000 7f000ffff640 1 -- 172.21.15.89:0/2059796483 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 598373243 0 0) 0x7f00181d4e60 con 0x7f00180b8240 2024-03-29T11:35:02.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.074+0000 7f000ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:02.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.074+0000 7f000ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:02.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.074+0000 7f000ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:02.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.074+0000 7f000ffff640 10 monclient(hunting): my global_id is 4153 2024-03-29T11:35:02.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.074+0000 7f000ffff640 1 -- 172.21.15.89:0/2059796483 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efff8004eb0 con 0x7f00180b8240 2024-03-29T11:35:02.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.074+0000 7f000ffff640 1 -- 172.21.15.89:0/2059796483 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1634168166 0 0) 0x7f0018131780 con 0x7f001809c8c0 2024-03-29T11:35:02.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.074+0000 7f000ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:02.086 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.074+0000 7f000ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:02.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.074+0000 7f000ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:02.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.074+0000 7f000ffff640 10 monclient(hunting): my global_id is 4149 2024-03-29T11:35:02.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.074+0000 7f000ffff640 1 -- 172.21.15.89:0/2059796483 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00181d4e60 con 0x7f001809c8c0 2024-03-29T11:35:02.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.074+0000 7f000ffff640 1 -- 172.21.15.89:0/2059796483 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3294584047 0 0) 0x7f00181d8040 con 0x7f0018146090 2024-03-29T11:35:02.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.074+0000 7f000ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:02.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.074+0000 7f000ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:02.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.074+0000 7f000ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:02.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.074+0000 7f000ffff640 10 monclient(hunting): my global_id is 4142 2024-03-29T11:35:02.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.074+0000 7f000ffff640 1 -- 172.21.15.89:0/2059796483 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0018131780 con 0x7f0018146090 2024-03-29T11:35:02.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f000ffff640 1 -- 172.21.15.89:0/2059796483 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2621931140 0 0) 0x7f00181d4e60 con 0x7f001809c8c0 2024-03-29T11:35:02.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f000ffff640 1 -- 172.21.15.89:0/2059796483 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00181d8040 con 0x7f001809c8c0 2024-03-29T11:35:02.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f000ffff640 1 -- 172.21.15.89:0/2059796483 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f0000001b80 con 0x7f001809c8c0 2024-03-29T11:35:02.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f000ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:02.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f000ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:35:02.087 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f000ffff640 10 monclient(hunting): dump: 2024-03-29T11:35:02.088 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:02.088 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:02.088 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:02.088 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:02.088 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:02.088 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:02.088 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:02.088 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:02.088 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:02.088 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:02.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f000ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:02.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f000ffff640 1 -- 172.21.15.89:0/2059796483 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2447686599 0 0) 0x7efff8004eb0 con 0x7f00180b8240 2024-03-29T11:35:02.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f000ffff640 1 -- 172.21.15.89:0/2059796483 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00181d4e60 con 0x7f00180b8240 2024-03-29T11:35:02.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f000ffff640 1 -- 172.21.15.89:0/2059796483 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 714489058 0 0) 0x7f0018131780 con 0x7f0018146090 2024-03-29T11:35:02.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f000ffff640 1 -- 172.21.15.89:0/2059796483 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efff8004eb0 con 0x7f0018146090 2024-03-29T11:35:02.088 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f000ffff640 1 -- 172.21.15.89:0/2059796483 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f0008001890 con 0x7f00180b8240 2024-03-29T11:35:02.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f000ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:02.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f000ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:02.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f000ffff640 10 monclient(hunting): dump: 2024-03-29T11:35:02.089 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:02.089 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:02.089 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:02.089 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:02.089 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:02.089 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:02.089 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:02.089 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:02.089 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:02.089 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:02.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f000ffff640 1 -- 172.21.15.89:0/2059796483 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f0010001c30 con 0x7f0018146090 2024-03-29T11:35:02.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f000ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:02.089 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f000ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:02.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f000ffff640 10 monclient(hunting): dump: 2024-03-29T11:35:02.090 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:02.090 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:02.090 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:02.090 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:02.090 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:02.090 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:02.090 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:02.090 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:02.090 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:02.090 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:02.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f000ffff640 1 -- 172.21.15.89:0/2059796483 <== mon.0 v1:172.21.15.72:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3433061345 0 0) 0x7f00181d8040 con 0x7f001809c8c0 2024-03-29T11:35:02.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f000ffff640 1 -- 172.21.15.89:0/2059796483 >> v1:172.21.15.72:6790/0 conn(0x7f0018146090 legacy=0x7f00181e1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:02.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f000ffff640 1 -- 172.21.15.89:0/2059796483 >> v1:172.21.15.89:6789/0 conn(0x7f00180b8240 legacy=0x7f00181d7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:02.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f000ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:35:02.090 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f000ffff640 1 monclient: found mon.a 2024-03-29T11:35:02.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f000ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:02.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f000ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:35:02.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f000ffff640 1 -- 172.21.15.89:0/2059796483 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0018132f10 con 0x7f001809c8c0 2024-03-29T11:35:02.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f000ffff640 10 monclient: _finish_auth 0 2024-03-29T11:35:02.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f000ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:35:02.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f000ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:02.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f001d8cf640 5 monclient: authenticate success, global_id 4149 2024-03-29T11:35:02.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f001d8cf640 10 monclient: _renew_subs 2024-03-29T11:35:02.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f001d8cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:35:02.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f001d8cf640 1 -- 172.21.15.89:0/2059796483 --> v1:172.21.15.72:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0018131b90 con 0x7f001809c8c0 2024-03-29T11:35:02.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f001d8cf640 10 client.4149.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:02.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f001d8cf640 10 monclient: _renew_subs 2024-03-29T11:35:02.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f001d8cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:35:02.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f001d8cf640 1 -- 172.21.15.89:0/2059796483 --> v1:172.21.15.72:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f00181e3330 con 0x7f001809c8c0 2024-03-29T11:35:02.091 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f001d8cf640 1 librados: init done 2024-03-29T11:35:02.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f000ffff640 1 -- 172.21.15.89:0/2059796483 <== mon.0 v1:172.21.15.72:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0000004a90 con 0x7f001809c8c0 2024-03-29T11:35:02.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f000ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:02.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f000ffff640 1 -- 172.21.15.89:0/2059796483 <== mon.0 v1:172.21.15.72:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f0000004da0 con 0x7f001809c8c0 2024-03-29T11:35:02.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f000ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:02.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f000ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:02.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f00157fa640 4 set_mon_vals no callback set 2024-03-29T11:35:02.092 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f000ffff640 10 monclient: dump: 2024-03-29T11:35:02.092 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:02.092 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:02.092 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:02.092 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:02.092 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:02.092 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:02.092 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:02.092 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:02.093 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:02.093 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:02.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f000ffff640 1 -- 172.21.15.89:0/2059796483 <== mon.0 v1:172.21.15.72:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0000005290 con 0x7f001809c8c0 2024-03-29T11:35:02.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f000ffff640 1 -- 172.21.15.89:0/2059796483 <== mon.0 v1:172.21.15.72:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 4280289281 0 0) 0x7f0000005b30 con 0x7f001809c8c0 2024-03-29T11:35:02.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f000ffff640 10 client.4149.objecter ms_dispatch 0x7f0018000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T11:35:02.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f000ffff640 3 client.4149.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T11:35:02.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f000ffff640 3 client.4149.objecter handle_osd_map decoding full epoch 8 2024-03-29T11:35:02.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.075+0000 7f000ffff640 20 client.4149.objecter dump_active .. 0 homeless 2024-03-29T11:35:02.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.076+0000 7f001d8cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:02.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.076+0000 7f001d8cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:02.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.076+0000 7f001d8cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:35:02.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.076+0000 7f001d8cf640 1 -- 172.21.15.89:0/2059796483 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efffc005080 con 0x7f001809c8c0 2024-03-29T11:35:02.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.078+0000 7f000ffff640 1 -- 172.21.15.89:0/2059796483 <== mon.0 v1:172.21.15.72:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0000005e10 con 0x7f001809c8c0 2024-03-29T11:35:02.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.078+0000 7f000ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:02.093 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.078+0000 7f000ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:02.142 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-29T11:35:02.144+0000 7fc853e3e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:35:02.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.151+0000 7f001d8cf640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "7c9b4262-0624-4322-b4be-78b105d0608e", "id": 6}] 2024-03-29T11:35:02.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.151+0000 7f001d8cf640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "7c9b4262-0624-4322-b4be-78b105d0608e", "id": 6}] 2024-03-29T11:35:02.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.151+0000 7f001d8cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:35:02.149 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.151+0000 7f001d8cf640 1 -- 172.21.15.89:0/2059796483 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "osd new", "uuid": "7c9b4262-0624-4322-b4be-78b105d0608e", "id": 6} v 0) v1 -- 0x7efffc0058b0 con 0x7f001809c8c0 2024-03-29T11:35:02.150 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.152+0000 7f000ffff640 1 -- 172.21.15.89:0/2059796483 <== mon.0 v1:172.21.15.72:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7c9b4262-0624-4322-b4be-78b105d0608e", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 1049464594 0 1113811849) 0x7f0000005610 con 0x7f001809c8c0 2024-03-29T11:35:02.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.152+0000 7f000ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "7c9b4262-0624-4322-b4be-78b105d0608e", "id": 6}] 2024-03-29T11:35:02.151 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.152+0000 7f000ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:02.151 INFO:teuthology.orchestra.run.smithi089.stdout:6 2024-03-29T11:35:02.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.153+0000 7f001d8cf640 10 librados: watch_flush enter 2024-03-29T11:35:02.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.153+0000 7f001d8cf640 10 librados: watch_flush exit 2024-03-29T11:35:02.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.153+0000 7f001d8cf640 20 client.4149.objecter shutdown clearing up homeless session... 2024-03-29T11:35:02.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.153+0000 7f001d8cf640 10 client.4149.objecter successfully canceled tick 2024-03-29T11:35:02.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.153+0000 7f001d8cf640 10 monclient: shutdown 2024-03-29T11:35:02.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.153+0000 7f001d8cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:02.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.153+0000 7f001d8cf640 1 -- 172.21.15.89:0/2059796483 >> v1:172.21.15.72:6789/0 conn(0x7f001809c8c0 legacy=0x7f0018131070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:02.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.153+0000 7f001d8cf640 1 -- 172.21.15.89:0/2059796483 shutdown_connections 2024-03-29T11:35:02.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.153+0000 7f001d8cf640 1 -- 172.21.15.89:0/2059796483 >> 172.21.15.89:0/2059796483 conn(0x7f0018066b00 msgr2=0x7f001813dd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:02.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.153+0000 7f001d8cf640 1 -- 172.21.15.89:0/2059796483 shutdown_connections 2024-03-29T11:35:02.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.153+0000 7f001d8cf640 1 -- 172.21.15.89:0/2059796483 wait complete. 2024-03-29T11:35:02.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.153+0000 7f001d8cf640 1 librados: shutdown 2024-03-29T11:35:02.161 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-29T11:35:02.163+0000 7f1f7fc2b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:35:02.162 DEBUG:teuthology.orchestra.run.smithi089:> sudo ceph --cluster ceph osd new 5491b1cb-cff3-4c28-803f-bc9c40c28b35 7 2024-03-29T11:35:02.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.263+0000 7fc09ee86640 10 monclient: get_monmap_and_config 2024-03-29T11:35:02.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.264+0000 7fc09ee86640 10 monclient: build_initial_monmap 2024-03-29T11:35:02.262 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.264+0000 7fc09ee86640 10 monclient: monmap: 2024-03-29T11:35:02.262 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 0 2024-03-29T11:35:02.262 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:02.262 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:35:02.265418+0000 2024-03-29T11:35:02.262 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:35:02.265418+0000 2024-03-29T11:35:02.262 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:02.262 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:02.263 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:35:02.263 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:35:02.263 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:35:02.263 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:02.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.264+0000 7fc09ee86640 1 Processor -- start 2024-03-29T11:35:02.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.264+0000 7fc09ee86640 1 -- start start 2024-03-29T11:35:02.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.264+0000 7fc09ee86640 10 monclient: init 2024-03-29T11:35:02.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.265+0000 7fc09ee86640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:02.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.265+0000 7fc09ee86640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:35:02.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.265+0000 7fc09ee86640 10 monclient(hunting): picked mon.noname-c con 0x7fc09809ac40 addr v1:172.21.15.89:6789/0 2024-03-29T11:35:02.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.265+0000 7fc09ee86640 10 monclient(hunting): picked mon.noname-a con 0x7fc09809c870 addr v1:172.21.15.72:6789/0 2024-03-29T11:35:02.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.265+0000 7fc09ee86640 10 monclient(hunting): picked mon.noname-b con 0x7fc0980b4360 addr v1:172.21.15.72:6790/0 2024-03-29T11:35:02.270 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.265+0000 7fc09ee86640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc09809dec0 con 0x7fc09809c870 2024-03-29T11:35:02.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.265+0000 7fc09ee86640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc09809da30 con 0x7fc09809ac40 2024-03-29T11:35:02.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.265+0000 7fc09ee86640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0980b39c0 con 0x7fc0980b4360 2024-03-29T11:35:02.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.265+0000 7fc09ee86640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:02.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.265+0000 7fc09ee86640 10 monclient(hunting): authenticate will time out at 874.538377s 2024-03-29T11:35:02.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.265+0000 7fc09cc22640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc09809ac40 0x7fc0980b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57106/0 (socket says 172.21.15.89:57106) 2024-03-29T11:35:02.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.265+0000 7fc09cc22640 1 -- 172.21.15.89:0/3667407975 learned_addr learned my addr 172.21.15.89:0/3667407975 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-03-29T11:35:02.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.265+0000 7fc0977fe640 1 -- 172.21.15.89:0/3667407975 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 42051360 0 0) 0x7fc09809da30 con 0x7fc09809ac40 2024-03-29T11:35:02.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.265+0000 7fc0977fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:02.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.265+0000 7fc0977fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:02.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.265+0000 7fc0977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:02.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.265+0000 7fc0977fe640 10 monclient(hunting): my global_id is 4156 2024-03-29T11:35:02.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.265+0000 7fc0977fe640 1 -- 172.21.15.89:0/3667407975 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc088003520 con 0x7fc09809ac40 2024-03-29T11:35:02.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.265+0000 7fc0977fe640 1 -- 172.21.15.89:0/3667407975 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1746253989 0 0) 0x7fc0980b39c0 con 0x7fc0980b4360 2024-03-29T11:35:02.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.265+0000 7fc0977fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:02.271 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.265+0000 7fc0977fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:02.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.265+0000 7fc0977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:02.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.265+0000 7fc0977fe640 10 monclient(hunting): my global_id is 4145 2024-03-29T11:35:02.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.265+0000 7fc0977fe640 1 -- 172.21.15.89:0/3667407975 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc09809da30 con 0x7fc0980b4360 2024-03-29T11:35:02.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.265+0000 7fc0977fe640 1 -- 172.21.15.89:0/3667407975 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3012998785 0 0) 0x7fc084001110 con 0x7fc09809c870 2024-03-29T11:35:02.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.265+0000 7fc0977fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:02.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.265+0000 7fc0977fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:02.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.265+0000 7fc0977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:02.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.265+0000 7fc0977fe640 10 monclient(hunting): my global_id is 4152 2024-03-29T11:35:02.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.265+0000 7fc0977fe640 1 -- 172.21.15.89:0/3667407975 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0980b39c0 con 0x7fc09809c870 2024-03-29T11:35:02.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.265+0000 7fc0977fe640 1 -- 172.21.15.89:0/3667407975 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1504853072 0 0) 0x7fc088003520 con 0x7fc09809ac40 2024-03-29T11:35:02.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.265+0000 7fc0977fe640 1 -- 172.21.15.89:0/3667407975 >> v1:172.21.15.72:6790/0 conn(0x7fc0980b4360 legacy=0x7fc09814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:02.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.265+0000 7fc0977fe640 1 -- 172.21.15.89:0/3667407975 >> v1:172.21.15.72:6789/0 conn(0x7fc09809c870 legacy=0x7fc09813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:02.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.265+0000 7fc0977fe640 10 monclient: _finish_hunting 0 2024-03-29T11:35:02.272 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.265+0000 7fc0977fe640 1 monclient: found mon.noname-c 2024-03-29T11:35:02.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.265+0000 7fc0977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:02.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.265+0000 7fc0977fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.89:6789/0 2024-03-29T11:35:02.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.266+0000 7fc0977fe640 1 -- 172.21.15.89:0/3667407975 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0980b5160 con 0x7fc09809ac40 2024-03-29T11:35:02.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.266+0000 7fc0977fe640 10 monclient: _finish_auth 0 2024-03-29T11:35:02.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.266+0000 7fc0977fe640 10 monclient: _check_auth_tickets 2024-03-29T11:35:02.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.266+0000 7fc0977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:02.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.266+0000 7fc0977fe640 1 -- 172.21.15.89:0/3667407975 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fc08c001690 con 0x7fc09809ac40 2024-03-29T11:35:02.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.266+0000 7fc09ee86640 5 monclient: authenticate success, global_id 4156 2024-03-29T11:35:02.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.266+0000 7fc09ee86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:02.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.266+0000 7fc0977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:02.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.266+0000 7fc0977fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:02.273 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.266+0000 7fc0977fe640 10 monclient: dump: 2024-03-29T11:35:02.273 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:02.273 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:02.273 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:02.273 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:02.274 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:02.274 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:02.274 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:02.274 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:02.274 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:02.274 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:02.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.266+0000 7fc09ee86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:02.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.266+0000 7fc0977fe640 1 -- 172.21.15.89:0/3667407975 <== mon.1 v1:172.21.15.89:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc08c001870 con 0x7fc09809ac40 2024-03-29T11:35:02.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.266+0000 7fc0977fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:02.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.266+0000 7fc0977fe640 1 -- 172.21.15.89:0/3667407975 <== mon.1 v1:172.21.15.89:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fc08c001c80 con 0x7fc09809ac40 2024-03-29T11:35:02.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.266+0000 7fc0977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:02.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.266+0000 7fc0977fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:02.274 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.266+0000 7fc0977fe640 10 monclient: dump: 2024-03-29T11:35:02.274 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:02.274 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:02.274 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:02.275 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:02.275 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:02.275 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:02.275 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:02.275 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:02.275 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:02.275 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:02.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.266+0000 7fc09ee86640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:02.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.266+0000 7fc09ee86640 4 set_mon_vals no callback set 2024-03-29T11:35:02.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.266+0000 7fc09ee86640 10 monclient: shutdown 2024-03-29T11:35:02.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.266+0000 7fc09ee86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:02.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.266+0000 7fc09ee86640 1 -- 172.21.15.89:0/3667407975 >> v1:172.21.15.89:6789/0 conn(0x7fc09809ac40 legacy=0x7fc0980b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:02.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.266+0000 7fc09ee86640 1 -- 172.21.15.89:0/3667407975 shutdown_connections 2024-03-29T11:35:02.275 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.266+0000 7fc09ee86640 1 -- 172.21.15.89:0/3667407975 >> 172.21.15.89:0/3667407975 conn(0x7fc098066b00 msgr2=0x7fc098066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:02.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.266+0000 7fc09ee86640 1 -- 172.21.15.89:0/3667407975 shutdown_connections 2024-03-29T11:35:02.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.266+0000 7fc09ee86640 1 -- 172.21.15.89:0/3667407975 wait complete. 2024-03-29T11:35:02.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.267+0000 7fc09ee86640 10 monclient: build_initial_monmap 2024-03-29T11:35:02.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.267+0000 7fc09ee86640 10 monclient: monmap: 2024-03-29T11:35:02.276 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 0 2024-03-29T11:35:02.276 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:02.276 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 0.000000 2024-03-29T11:35:02.276 INFO:teuthology.orchestra.run.smithi089.stderr:created 0.000000 2024-03-29T11:35:02.276 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:02.276 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:02.276 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:35:02.276 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:35:02.276 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:35:02.276 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:02.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.267+0000 7fc09ee86640 1 librados: starting msgr at 2024-03-29T11:35:02.276 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.267+0000 7fc09ee86640 1 librados: starting objecter 2024-03-29T11:35:02.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.267+0000 7fc09ee86640 1 Processor -- start 2024-03-29T11:35:02.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.267+0000 7fc09ee86640 1 -- start start 2024-03-29T11:35:02.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.267+0000 7fc09ee86640 1 librados: setting wanted keys 2024-03-29T11:35:02.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.267+0000 7fc09ee86640 1 librados: calling monclient init 2024-03-29T11:35:02.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.267+0000 7fc09ee86640 10 monclient: init 2024-03-29T11:35:02.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc09ee86640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:02.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc09ee86640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:35:02.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc09ee86640 10 monclient(hunting): picked mon.noname-a con 0x7fc09809ac40 addr v1:172.21.15.72:6789/0 2024-03-29T11:35:02.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc09ee86640 10 monclient(hunting): picked mon.noname-b con 0x7fc098138f60 addr v1:172.21.15.89:6789/0 2024-03-29T11:35:02.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc09ee86640 10 monclient(hunting): picked mon.noname-c con 0x7fc0980afdc0 addr v1:172.21.15.72:6790/0 2024-03-29T11:35:02.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc09ee86640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0981db610 con 0x7fc09809ac40 2024-03-29T11:35:02.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc09ee86640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc098139a00 con 0x7fc098138f60 2024-03-29T11:35:02.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc09ee86640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0980b0230 con 0x7fc0980afdc0 2024-03-29T11:35:02.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc09ee86640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:02.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc09ee86640 10 monclient(hunting): authenticate will time out at 874.541373s 2024-03-29T11:35:02.277 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc097fff640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc098138f60 0x7fc0980af6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.89:57112/0 (socket says 172.21.15.89:57112) 2024-03-29T11:35:02.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc097fff640 1 -- 172.21.15.89:0/1515706958 learned_addr learned my addr 172.21.15.89:0/1515706958 (peer_addr_for_me v1:172.21.15.89:0/0) 2024-03-29T11:35:02.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc0957fa640 10 client.?.objecter ms_handle_connect 0x7fc098138f60 2024-03-29T11:35:02.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc0957fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:02.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc0957fa640 10 client.?.objecter ms_handle_connect 0x7fc0980afdc0 2024-03-29T11:35:02.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc0957fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:02.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc0957fa640 10 client.?.objecter ms_handle_connect 0x7fc09809ac40 2024-03-29T11:35:02.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc0957fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:02.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc0957fa640 1 -- 172.21.15.89:0/1515706958 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3898052075 0 0) 0x7fc098139a00 con 0x7fc098138f60 2024-03-29T11:35:02.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc0957fa640 10 monclient(hunting): _negotiate 2024-03-29T11:35:02.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc0957fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:02.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc0957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:02.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc0957fa640 10 monclient(hunting): my global_id is 4159 2024-03-29T11:35:02.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc0957fa640 1 -- 172.21.15.89:0/1515706958 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc07c004eb0 con 0x7fc098138f60 2024-03-29T11:35:02.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc0957fa640 1 -- 172.21.15.89:0/1515706958 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1693041097 0 0) 0x7fc0980b0230 con 0x7fc0980afdc0 2024-03-29T11:35:02.278 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc0957fa640 10 monclient(hunting): _negotiate 2024-03-29T11:35:02.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc0957fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:02.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc0957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:02.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc0957fa640 10 monclient(hunting): my global_id is 4148 2024-03-29T11:35:02.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc0957fa640 1 -- 172.21.15.89:0/1515706958 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc098139a00 con 0x7fc0980afdc0 2024-03-29T11:35:02.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc0957fa640 1 -- 172.21.15.89:0/1515706958 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2637274577 0 0) 0x7fc0981db610 con 0x7fc09809ac40 2024-03-29T11:35:02.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc0957fa640 10 monclient(hunting): _negotiate 2024-03-29T11:35:02.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc0957fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:02.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc0957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:02.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc0957fa640 10 monclient(hunting): my global_id is 4155 2024-03-29T11:35:02.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc0957fa640 1 -- 172.21.15.89:0/1515706958 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0980b0230 con 0x7fc09809ac40 2024-03-29T11:35:02.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc0957fa640 1 -- 172.21.15.89:0/1515706958 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1918752673 0 0) 0x7fc098139a00 con 0x7fc0980afdc0 2024-03-29T11:35:02.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc0957fa640 1 -- 172.21.15.89:0/1515706958 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0981db610 con 0x7fc0980afdc0 2024-03-29T11:35:02.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc0957fa640 1 -- 172.21.15.89:0/1515706958 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fc090001a20 con 0x7fc0980afdc0 2024-03-29T11:35:02.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc0957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:02.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc0957fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:02.279 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc0957fa640 10 monclient(hunting): dump: 2024-03-29T11:35:02.280 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:02.280 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:02.280 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:02.280 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:02.280 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:02.280 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:02.280 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:02.280 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:02.280 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:02.280 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:02.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc0957fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:02.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc0957fa640 1 -- 172.21.15.89:0/1515706958 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 550955837 0 0) 0x7fc07c004eb0 con 0x7fc098138f60 2024-03-29T11:35:02.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc0957fa640 1 -- 172.21.15.89:0/1515706958 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc098139a00 con 0x7fc098138f60 2024-03-29T11:35:02.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc0957fa640 1 -- 172.21.15.89:0/1515706958 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fc084001860 con 0x7fc098138f60 2024-03-29T11:35:02.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc0957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:02.280 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc0957fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:02.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc0957fa640 10 monclient(hunting): dump: 2024-03-29T11:35:02.281 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:02.281 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:02.281 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:02.281 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:02.281 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:02.281 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:02.281 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:02.281 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:02.281 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:02.281 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:02.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc0957fa640 1 -- 172.21.15.89:0/1515706958 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2613278274 0 0) 0x7fc0980b0230 con 0x7fc09809ac40 2024-03-29T11:35:02.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.268+0000 7fc0957fa640 1 -- 172.21.15.89:0/1515706958 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc07c004eb0 con 0x7fc09809ac40 2024-03-29T11:35:02.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.269+0000 7fc0957fa640 1 -- 172.21.15.89:0/1515706958 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fc08c0019e0 con 0x7fc09809ac40 2024-03-29T11:35:02.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.269+0000 7fc0957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:02.281 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.269+0000 7fc0957fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:02.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.269+0000 7fc0957fa640 10 monclient(hunting): dump: 2024-03-29T11:35:02.282 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:02.282 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:02.282 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:02.282 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:02.282 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:02.282 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:02.282 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:02.282 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:02.282 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:02.282 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:02.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.269+0000 7fc0957fa640 1 -- 172.21.15.89:0/1515706958 <== mon.2 v1:172.21.15.72:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3372266492 0 0) 0x7fc0981db610 con 0x7fc0980afdc0 2024-03-29T11:35:02.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.269+0000 7fc0957fa640 1 -- 172.21.15.89:0/1515706958 >> v1:172.21.15.89:6789/0 conn(0x7fc098138f60 legacy=0x7fc0980af6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:02.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.269+0000 7fc0957fa640 1 -- 172.21.15.89:0/1515706958 >> v1:172.21.15.72:6789/0 conn(0x7fc09809ac40 legacy=0x7fc0981dac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:02.282 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.269+0000 7fc0957fa640 10 monclient: _finish_hunting 0 2024-03-29T11:35:02.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.269+0000 7fc0957fa640 1 monclient: found mon.c 2024-03-29T11:35:02.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.269+0000 7fc0957fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:02.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.269+0000 7fc0957fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:35:02.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.269+0000 7fc0957fa640 1 -- 172.21.15.89:0/1515706958 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0980b03c0 con 0x7fc0980afdc0 2024-03-29T11:35:02.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.269+0000 7fc0957fa640 10 monclient: _finish_auth 0 2024-03-29T11:35:02.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.269+0000 7fc0957fa640 10 monclient: _check_auth_tickets 2024-03-29T11:35:02.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.269+0000 7fc0957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:02.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.269+0000 7fc09ee86640 5 monclient: authenticate success, global_id 4148 2024-03-29T11:35:02.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.269+0000 7fc09ee86640 10 monclient: _renew_subs 2024-03-29T11:35:02.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.269+0000 7fc09ee86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:35:02.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.269+0000 7fc09ee86640 1 -- 172.21.15.89:0/1515706958 --> v1:172.21.15.72:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0981e36e0 con 0x7fc0980afdc0 2024-03-29T11:35:02.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.269+0000 7fc09ee86640 10 client.4148.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:02.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.269+0000 7fc09ee86640 10 monclient: _renew_subs 2024-03-29T11:35:02.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.269+0000 7fc09ee86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:35:02.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.269+0000 7fc09ee86640 1 -- 172.21.15.89:0/1515706958 --> v1:172.21.15.72:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0981e3a50 con 0x7fc0980afdc0 2024-03-29T11:35:02.283 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.269+0000 7fc09ee86640 1 librados: init done 2024-03-29T11:35:02.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.269+0000 7fc0957fa640 1 -- 172.21.15.89:0/1515706958 <== mon.2 v1:172.21.15.72:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc090001f50 con 0x7fc0980afdc0 2024-03-29T11:35:02.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.269+0000 7fc0957fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:02.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.269+0000 7fc0957fa640 1 -- 172.21.15.89:0/1515706958 <== mon.2 v1:172.21.15.72:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fc090003970 con 0x7fc0980afdc0 2024-03-29T11:35:02.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.269+0000 7fc0957fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:02.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.269+0000 7fc0957fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:02.284 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.269+0000 7fc0957fa640 10 monclient: dump: 2024-03-29T11:35:02.284 INFO:teuthology.orchestra.run.smithi089.stderr:epoch 1 2024-03-29T11:35:02.284 INFO:teuthology.orchestra.run.smithi089.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:02.284 INFO:teuthology.orchestra.run.smithi089.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:02.284 INFO:teuthology.orchestra.run.smithi089.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:02.284 INFO:teuthology.orchestra.run.smithi089.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:02.284 INFO:teuthology.orchestra.run.smithi089.stderr:election_strategy: 1 2024-03-29T11:35:02.284 INFO:teuthology.orchestra.run.smithi089.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:02.284 INFO:teuthology.orchestra.run.smithi089.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:02.284 INFO:teuthology.orchestra.run.smithi089.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:02.284 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:35:02.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.269+0000 7fc096ffd640 4 set_mon_vals no callback set 2024-03-29T11:35:02.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.269+0000 7fc0957fa640 1 -- 172.21.15.89:0/1515706958 <== mon.2 v1:172.21.15.72:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc090003e60 con 0x7fc0980afdc0 2024-03-29T11:35:02.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.269+0000 7fc0957fa640 1 -- 172.21.15.89:0/1515706958 <== mon.2 v1:172.21.15.72:6790/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 3457389908 0 0) 0x7fc090005670 con 0x7fc0980afdc0 2024-03-29T11:35:02.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.269+0000 7fc0957fa640 10 client.4148.objecter ms_dispatch 0x7fc098000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T11:35:02.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.269+0000 7fc0957fa640 3 client.4148.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T11:35:02.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.269+0000 7fc0957fa640 3 client.4148.objecter handle_osd_map decoding full epoch 9 2024-03-29T11:35:02.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.269+0000 7fc0957fa640 20 client.4148.objecter dump_active .. 0 homeless 2024-03-29T11:35:02.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.270+0000 7fc09ee86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:02.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.270+0000 7fc09ee86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:02.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.270+0000 7fc09ee86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:35:02.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.270+0000 7fc09ee86640 1 -- 172.21.15.89:0/1515706958 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc078005080 con 0x7fc0980afdc0 2024-03-29T11:35:02.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.272+0000 7fc0957fa640 1 -- 172.21.15.89:0/1515706958 <== mon.2 v1:172.21.15.72:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fc090005070 con 0x7fc0980afdc0 2024-03-29T11:35:02.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.272+0000 7fc0957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:02.285 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.272+0000 7fc0957fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:02.286 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-29T11:35:02.278+0000 7fc853e3e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:35:02.303 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-29T11:35:02.305+0000 7f1f7fc2b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:35:02.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.345+0000 7fc09ee86640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5491b1cb-cff3-4c28-803f-bc9c40c28b35", "id": 7}] 2024-03-29T11:35:02.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.345+0000 7fc09ee86640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5491b1cb-cff3-4c28-803f-bc9c40c28b35", "id": 7}] 2024-03-29T11:35:02.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.345+0000 7fc09ee86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:35:02.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.345+0000 7fc09ee86640 1 -- 172.21.15.89:0/1515706958 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "osd new", "uuid": "5491b1cb-cff3-4c28-803f-bc9c40c28b35", "id": 7} v 0) v1 -- 0x7fc0780058b0 con 0x7fc0980afdc0 2024-03-29T11:35:02.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.348+0000 7fc0957fa640 1 -- 172.21.15.89:0/1515706958 <== mon.2 v1:172.21.15.72:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5491b1cb-cff3-4c28-803f-bc9c40c28b35", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 2846094853 0 1371665406) 0x7fc090005360 con 0x7fc0980afdc0 2024-03-29T11:35:02.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.348+0000 7fc0957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5491b1cb-cff3-4c28-803f-bc9c40c28b35", "id": 7}] 2024-03-29T11:35:02.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.348+0000 7fc0957fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:02.346 INFO:teuthology.orchestra.run.smithi089.stdout:7 2024-03-29T11:35:02.347 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.348+0000 7fc09ee86640 10 librados: watch_flush enter 2024-03-29T11:35:02.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.349+0000 7fc09ee86640 10 librados: watch_flush exit 2024-03-29T11:35:02.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.349+0000 7fc09ee86640 20 client.4148.objecter shutdown clearing up homeless session... 2024-03-29T11:35:02.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.349+0000 7fc09ee86640 10 client.4148.objecter successfully canceled tick 2024-03-29T11:35:02.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.349+0000 7fc09ee86640 10 monclient: shutdown 2024-03-29T11:35:02.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.349+0000 7fc09ee86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:02.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.349+0000 7fc09ee86640 1 -- 172.21.15.89:0/1515706958 >> v1:172.21.15.72:6790/0 conn(0x7fc0980afdc0 legacy=0x7fc0981e1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:02.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.349+0000 7fc09ee86640 1 -- 172.21.15.89:0/1515706958 shutdown_connections 2024-03-29T11:35:02.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.349+0000 7fc09ee86640 1 -- 172.21.15.89:0/1515706958 >> 172.21.15.89:0/1515706958 conn(0x7fc098066b00 msgr2=0x7fc0980b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:02.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.349+0000 7fc09ee86640 1 -- 172.21.15.89:0/1515706958 shutdown_connections 2024-03-29T11:35:02.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.349+0000 7fc09ee86640 1 -- 172.21.15.89:0/1515706958 wait complete. 2024-03-29T11:35:02.348 INFO:teuthology.orchestra.run.smithi089.stderr:2024-03-29T11:35:02.349+0000 7fc09ee86640 1 librados: shutdown 2024-03-29T11:35:02.357 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T11:35:02.357 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T11:35:02.359 INFO:tasks.ceph.osd.0:Started 2024-03-29T11:35:02.359 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T11:35:02.360 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T11:35:02.362 INFO:tasks.ceph.osd.1:Started 2024-03-29T11:35:02.362 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T11:35:02.362 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T11:35:02.364 INFO:tasks.ceph.osd.2:Started 2024-03-29T11:35:02.364 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T11:35:02.364 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T11:35:02.366 INFO:tasks.ceph.osd.3:Started 2024-03-29T11:35:02.366 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T11:35:02.366 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T11:35:02.368 INFO:tasks.ceph.osd.4:Started 2024-03-29T11:35:02.368 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T11:35:02.368 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T11:35:02.370 INFO:tasks.ceph.osd.5:Started 2024-03-29T11:35:02.371 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T11:35:02.371 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T11:35:02.373 INFO:tasks.ceph.osd.6:Started 2024-03-29T11:35:02.373 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T11:35:02.373 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T11:35:02.375 INFO:tasks.ceph.osd.7:Started 2024-03-29T11:35:02.375 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:02.404 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-29T11:35:02.407+0000 7fc853e3e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:35:02.430 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-29T11:35:02.432+0000 7f1f7fc2b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:35:02.468 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-29T11:35:02.470+0000 7fc853e3e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:35:02.488 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-29T11:35:02.490+0000 7f1f7fc2b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:35:02.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.495+0000 7f86ec42a640 10 monclient: get_monmap_and_config 2024-03-29T11:35:02.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.495+0000 7f86ec42a640 10 monclient: build_initial_monmap 2024-03-29T11:35:02.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.495+0000 7f86ec42a640 10 monclient: monmap: 2024-03-29T11:35:02.493 INFO:teuthology.orchestra.run.smithi072.stderr:epoch 0 2024-03-29T11:35:02.493 INFO:teuthology.orchestra.run.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:02.493 INFO:teuthology.orchestra.run.smithi072.stderr:last_changed 2024-03-29T11:35:02.496923+0000 2024-03-29T11:35:02.494 INFO:teuthology.orchestra.run.smithi072.stderr:created 2024-03-29T11:35:02.496923+0000 2024-03-29T11:35:02.494 INFO:teuthology.orchestra.run.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:02.494 INFO:teuthology.orchestra.run.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:02.494 INFO:teuthology.orchestra.run.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:35:02.494 INFO:teuthology.orchestra.run.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:35:02.494 INFO:teuthology.orchestra.run.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:35:02.494 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-03-29T11:35:02.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.496+0000 7f86ec42a640 1 Processor -- start 2024-03-29T11:35:02.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.496+0000 7f86ec42a640 1 -- start start 2024-03-29T11:35:02.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.496+0000 7f86ec42a640 10 monclient: init 2024-03-29T11:35:02.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.497+0000 7f86ec42a640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:02.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.497+0000 7f86ec42a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:35:02.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.497+0000 7f86ec42a640 10 monclient(hunting): picked mon.noname-a con 0x7f86e40b8240 addr v1:172.21.15.72:6789/0 2024-03-29T11:35:02.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.497+0000 7f86ec42a640 10 monclient(hunting): picked mon.noname-c con 0x7f86e40b1f60 addr v1:172.21.15.89:6789/0 2024-03-29T11:35:02.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.497+0000 7f86ec42a640 10 monclient(hunting): picked mon.noname-b con 0x7f86e409c8c0 addr v1:172.21.15.72:6790/0 2024-03-29T11:35:02.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.497+0000 7f86ec42a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86e409bd80 con 0x7f86e40b8240 2024-03-29T11:35:02.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.497+0000 7f86ec42a640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86e409b960 con 0x7f86e40b1f60 2024-03-29T11:35:02.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.497+0000 7f86ec42a640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86e40b2ec0 con 0x7f86e409c8c0 2024-03-29T11:35:02.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.497+0000 7f86ec42a640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:02.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.497+0000 7f86ec42a640 10 monclient(hunting): authenticate will time out at 861.785438s 2024-03-29T11:35:02.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.497+0000 7f86ea1c6640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f86e40b8240 0x7f86e413f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:53688/0 (socket says 172.21.15.72:53688) 2024-03-29T11:35:02.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.497+0000 7f86ea1c6640 1 -- 172.21.15.72:0/383084216 learned_addr learned my addr 172.21.15.72:0/383084216 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:35:02.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.497+0000 7f86e91c4640 1 -- 172.21.15.72:0/383084216 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4095004942 0 0) 0x7f86e409bd80 con 0x7f86e40b8240 2024-03-29T11:35:02.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.497+0000 7f86e91c4640 10 monclient(hunting): _negotiate 2024-03-29T11:35:02.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.497+0000 7f86e91c4640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:02.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.497+0000 7f86e91c4640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:02.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.497+0000 7f86e91c4640 10 monclient(hunting): my global_id is 4182 2024-03-29T11:35:02.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.497+0000 7f86e91c4640 1 -- 172.21.15.72:0/383084216 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86d0003520 con 0x7f86e40b8240 2024-03-29T11:35:02.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.497+0000 7f86e91c4640 1 -- 172.21.15.72:0/383084216 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3560652303 0 0) 0x7f86e40b2ec0 con 0x7f86e409c8c0 2024-03-29T11:35:02.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.497+0000 7f86e91c4640 10 monclient(hunting): _negotiate 2024-03-29T11:35:02.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.497+0000 7f86e91c4640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:02.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.497+0000 7f86e91c4640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:02.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.497+0000 7f86e91c4640 10 monclient(hunting): my global_id is 4175 2024-03-29T11:35:02.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.497+0000 7f86e91c4640 1 -- 172.21.15.72:0/383084216 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86e409bd80 con 0x7f86e409c8c0 2024-03-29T11:35:02.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.497+0000 7f86e91c4640 1 -- 172.21.15.72:0/383084216 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1106857350 0 0) 0x7f86e409b960 con 0x7f86e40b1f60 2024-03-29T11:35:02.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.497+0000 7f86e91c4640 10 monclient(hunting): _negotiate 2024-03-29T11:35:02.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.497+0000 7f86e91c4640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:02.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.497+0000 7f86e91c4640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:02.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.497+0000 7f86e91c4640 10 monclient(hunting): my global_id is 4186 2024-03-29T11:35:02.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.497+0000 7f86e91c4640 1 -- 172.21.15.72:0/383084216 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86e40b2ec0 con 0x7f86e40b1f60 2024-03-29T11:35:02.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.497+0000 7f86e91c4640 1 -- 172.21.15.72:0/383084216 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1831473727 0 0) 0x7f86d0003520 con 0x7f86e40b8240 2024-03-29T11:35:02.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.498+0000 7f86e91c4640 1 -- 172.21.15.72:0/383084216 >> v1:172.21.15.72:6790/0 conn(0x7f86e409c8c0 legacy=0x7f86e414d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:02.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.498+0000 7f86e91c4640 1 -- 172.21.15.72:0/383084216 >> v1:172.21.15.89:6789/0 conn(0x7f86e40b1f60 legacy=0x7f86e409c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:02.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.498+0000 7f86e91c4640 10 monclient: _finish_hunting 0 2024-03-29T11:35:02.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.498+0000 7f86e91c4640 1 monclient: found mon.noname-a 2024-03-29T11:35:02.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.498+0000 7f86e91c4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:02.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.498+0000 7f86e91c4640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.72:6789/0 2024-03-29T11:35:02.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.498+0000 7f86e91c4640 1 -- 172.21.15.72:0/383084216 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86e413fb60 con 0x7f86e40b8240 2024-03-29T11:35:02.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.498+0000 7f86e91c4640 10 monclient: _finish_auth 0 2024-03-29T11:35:02.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.498+0000 7f86e91c4640 10 monclient: _check_auth_tickets 2024-03-29T11:35:02.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.498+0000 7f86e91c4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:02.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.498+0000 7f86e91c4640 1 -- 172.21.15.72:0/383084216 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f86d4001690 con 0x7f86e40b8240 2024-03-29T11:35:02.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.498+0000 7f86ec42a640 5 monclient: authenticate success, global_id 4182 2024-03-29T11:35:02.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.498+0000 7f86ec42a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:02.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.498+0000 7f86e91c4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:02.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.498+0000 7f86e91c4640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:35:02.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.498+0000 7f86e91c4640 10 monclient: dump: 2024-03-29T11:35:02.505 INFO:teuthology.orchestra.run.smithi072.stderr:epoch 1 2024-03-29T11:35:02.505 INFO:teuthology.orchestra.run.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:02.505 INFO:teuthology.orchestra.run.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:02.505 INFO:teuthology.orchestra.run.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:02.506 INFO:teuthology.orchestra.run.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:02.506 INFO:teuthology.orchestra.run.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:02.506 INFO:teuthology.orchestra.run.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:02.506 INFO:teuthology.orchestra.run.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:02.506 INFO:teuthology.orchestra.run.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:02.506 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-03-29T11:35:02.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.498+0000 7f86ec42a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:02.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.498+0000 7f86e91c4640 1 -- 172.21.15.72:0/383084216 <== mon.0 v1:172.21.15.72:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86d40017f0 con 0x7f86e40b8240 2024-03-29T11:35:02.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.498+0000 7f86e91c4640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:02.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.498+0000 7f86e91c4640 1 -- 172.21.15.72:0/383084216 <== mon.0 v1:172.21.15.72:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f86d4001c00 con 0x7f86e40b8240 2024-03-29T11:35:02.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.498+0000 7f86ec42a640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:02.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.498+0000 7f86ec42a640 4 set_mon_vals no callback set 2024-03-29T11:35:02.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.498+0000 7f86e91c4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:02.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.498+0000 7f86e91c4640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:02.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.498+0000 7f86e91c4640 10 monclient: dump: 2024-03-29T11:35:02.507 INFO:teuthology.orchestra.run.smithi072.stderr:epoch 1 2024-03-29T11:35:02.507 INFO:teuthology.orchestra.run.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:02.507 INFO:teuthology.orchestra.run.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:02.507 INFO:teuthology.orchestra.run.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:02.507 INFO:teuthology.orchestra.run.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:02.507 INFO:teuthology.orchestra.run.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:02.507 INFO:teuthology.orchestra.run.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:02.507 INFO:teuthology.orchestra.run.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:02.507 INFO:teuthology.orchestra.run.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:02.507 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-03-29T11:35:02.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.498+0000 7f86ec42a640 10 monclient: shutdown 2024-03-29T11:35:02.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.498+0000 7f86ec42a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:02.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.498+0000 7f86ec42a640 1 -- 172.21.15.72:0/383084216 >> v1:172.21.15.72:6789/0 conn(0x7f86e40b8240 legacy=0x7f86e413f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:02.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.498+0000 7f86ec42a640 1 -- 172.21.15.72:0/383084216 shutdown_connections 2024-03-29T11:35:02.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.498+0000 7f86ec42a640 1 -- 172.21.15.72:0/383084216 >> 172.21.15.72:0/383084216 conn(0x7f86e4066b00 msgr2=0x7f86e4066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:02.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.498+0000 7f86ec42a640 1 -- 172.21.15.72:0/383084216 shutdown_connections 2024-03-29T11:35:02.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.498+0000 7f86ec42a640 1 -- 172.21.15.72:0/383084216 wait complete. 2024-03-29T11:35:02.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.499+0000 7f86ec42a640 10 monclient: build_initial_monmap 2024-03-29T11:35:02.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.499+0000 7f86ec42a640 10 monclient: monmap: 2024-03-29T11:35:02.508 INFO:teuthology.orchestra.run.smithi072.stderr:epoch 0 2024-03-29T11:35:02.508 INFO:teuthology.orchestra.run.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:02.508 INFO:teuthology.orchestra.run.smithi072.stderr:last_changed 0.000000 2024-03-29T11:35:02.508 INFO:teuthology.orchestra.run.smithi072.stderr:created 0.000000 2024-03-29T11:35:02.508 INFO:teuthology.orchestra.run.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:02.508 INFO:teuthology.orchestra.run.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:02.508 INFO:teuthology.orchestra.run.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:35:02.508 INFO:teuthology.orchestra.run.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:35:02.508 INFO:teuthology.orchestra.run.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:35:02.508 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-03-29T11:35:02.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.499+0000 7f86ec42a640 1 librados: starting msgr at 2024-03-29T11:35:02.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.499+0000 7f86ec42a640 1 librados: starting objecter 2024-03-29T11:35:02.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.499+0000 7f86ec42a640 1 Processor -- start 2024-03-29T11:35:02.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.499+0000 7f86ec42a640 1 -- start start 2024-03-29T11:35:02.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.499+0000 7f86ec42a640 1 librados: setting wanted keys 2024-03-29T11:35:02.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.499+0000 7f86ec42a640 1 librados: calling monclient init 2024-03-29T11:35:02.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.499+0000 7f86ec42a640 10 monclient: init 2024-03-29T11:35:02.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86ec42a640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:02.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86ec42a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:35:02.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86ec42a640 10 monclient(hunting): picked mon.noname-b con 0x7f86e409c8c0 addr v1:172.21.15.89:6789/0 2024-03-29T11:35:02.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86ec42a640 10 monclient(hunting): picked mon.noname-a con 0x7f86e40b8240 addr v1:172.21.15.72:6789/0 2024-03-29T11:35:02.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86ec42a640 10 monclient(hunting): picked mon.noname-c con 0x7f86e4141e40 addr v1:172.21.15.72:6790/0 2024-03-29T11:35:02.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86e99c5640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f86e40b8240 0x7f86e41455e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:53702/0 (socket says 172.21.15.72:53702) 2024-03-29T11:35:02.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86e99c5640 1 -- 172.21.15.72:0/3656963608 learned_addr learned my addr 172.21.15.72:0/3656963608 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:35:02.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86ec42a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86e41314b0 con 0x7f86e40b8240 2024-03-29T11:35:02.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86ec42a640 1 -- 172.21.15.72:0/3656963608 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86e4131700 con 0x7f86e409c8c0 2024-03-29T11:35:02.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86ec42a640 1 -- 172.21.15.72:0/3656963608 --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86e41450c0 con 0x7f86e4141e40 2024-03-29T11:35:02.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86daffd640 10 client.?.objecter ms_handle_connect 0x7f86e40b8240 2024-03-29T11:35:02.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86daffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:02.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86ec42a640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:02.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86ec42a640 10 monclient(hunting): authenticate will time out at 861.788562s 2024-03-29T11:35:02.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86daffd640 10 client.?.objecter ms_handle_connect 0x7f86e4141e40 2024-03-29T11:35:02.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86daffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:02.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86daffd640 10 client.?.objecter ms_handle_connect 0x7f86e409c8c0 2024-03-29T11:35:02.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86daffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:02.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86daffd640 1 -- 172.21.15.72:0/3656963608 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3195783894 0 0) 0x7f86e41314b0 con 0x7f86e40b8240 2024-03-29T11:35:02.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86daffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:02.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86daffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:02.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:02.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86daffd640 10 monclient(hunting): my global_id is 4185 2024-03-29T11:35:02.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86daffd640 1 -- 172.21.15.72:0/3656963608 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86c8004eb0 con 0x7f86e40b8240 2024-03-29T11:35:02.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86daffd640 1 -- 172.21.15.72:0/3656963608 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1105410728 0 0) 0x7f86e41450c0 con 0x7f86e4141e40 2024-03-29T11:35:02.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86daffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:02.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86daffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:02.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:02.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86daffd640 10 monclient(hunting): my global_id is 4178 2024-03-29T11:35:02.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86daffd640 1 -- 172.21.15.72:0/3656963608 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86e41314b0 con 0x7f86e4141e40 2024-03-29T11:35:02.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86daffd640 1 -- 172.21.15.72:0/3656963608 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1319820296 0 0) 0x7f86e4131700 con 0x7f86e409c8c0 2024-03-29T11:35:02.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86daffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:02.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86daffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:02.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:02.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86daffd640 10 monclient(hunting): my global_id is 4189 2024-03-29T11:35:02.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86daffd640 1 -- 172.21.15.72:0/3656963608 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86e41450c0 con 0x7f86e409c8c0 2024-03-29T11:35:02.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86daffd640 1 -- 172.21.15.72:0/3656963608 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2108058107 0 0) 0x7f86c8004eb0 con 0x7f86e40b8240 2024-03-29T11:35:02.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86daffd640 1 -- 172.21.15.72:0/3656963608 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86e4131700 con 0x7f86e40b8240 2024-03-29T11:35:02.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86daffd640 1 -- 172.21.15.72:0/3656963608 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f86e00019d0 con 0x7f86e40b8240 2024-03-29T11:35:02.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:02.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86daffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:35:02.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86daffd640 10 monclient(hunting): dump: 2024-03-29T11:35:02.512 INFO:teuthology.orchestra.run.smithi072.stderr:epoch 1 2024-03-29T11:35:02.512 INFO:teuthology.orchestra.run.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:02.512 INFO:teuthology.orchestra.run.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:02.512 INFO:teuthology.orchestra.run.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:02.512 INFO:teuthology.orchestra.run.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:02.512 INFO:teuthology.orchestra.run.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:02.512 INFO:teuthology.orchestra.run.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:02.512 INFO:teuthology.orchestra.run.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:02.512 INFO:teuthology.orchestra.run.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:02.512 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-03-29T11:35:02.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.500+0000 7f86daffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:02.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86daffd640 1 -- 172.21.15.72:0/3656963608 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1204060761 0 0) 0x7f86e41450c0 con 0x7f86e409c8c0 2024-03-29T11:35:02.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86daffd640 1 -- 172.21.15.72:0/3656963608 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86c8004eb0 con 0x7f86e409c8c0 2024-03-29T11:35:02.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86daffd640 1 -- 172.21.15.72:0/3656963608 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f86d4001b00 con 0x7f86e409c8c0 2024-03-29T11:35:02.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:02.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86daffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:02.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86daffd640 10 monclient(hunting): dump: 2024-03-29T11:35:02.513 INFO:teuthology.orchestra.run.smithi072.stderr:epoch 1 2024-03-29T11:35:02.513 INFO:teuthology.orchestra.run.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:02.513 INFO:teuthology.orchestra.run.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:02.513 INFO:teuthology.orchestra.run.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:02.513 INFO:teuthology.orchestra.run.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:02.513 INFO:teuthology.orchestra.run.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:02.513 INFO:teuthology.orchestra.run.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:02.513 INFO:teuthology.orchestra.run.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:02.513 INFO:teuthology.orchestra.run.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:02.513 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-03-29T11:35:02.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86daffd640 1 -- 172.21.15.72:0/3656963608 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2705005722 0 0) 0x7f86e41314b0 con 0x7f86e4141e40 2024-03-29T11:35:02.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86daffd640 1 -- 172.21.15.72:0/3656963608 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86e41450c0 con 0x7f86e4141e40 2024-03-29T11:35:02.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86daffd640 1 -- 172.21.15.72:0/3656963608 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f86dc001b50 con 0x7f86e4141e40 2024-03-29T11:35:02.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:02.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86daffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:02.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86daffd640 10 monclient(hunting): dump: 2024-03-29T11:35:02.514 INFO:teuthology.orchestra.run.smithi072.stderr:epoch 1 2024-03-29T11:35:02.514 INFO:teuthology.orchestra.run.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:02.514 INFO:teuthology.orchestra.run.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:02.514 INFO:teuthology.orchestra.run.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:02.514 INFO:teuthology.orchestra.run.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:02.514 INFO:teuthology.orchestra.run.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:02.514 INFO:teuthology.orchestra.run.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:02.514 INFO:teuthology.orchestra.run.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:02.514 INFO:teuthology.orchestra.run.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:02.514 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-03-29T11:35:02.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86daffd640 1 -- 172.21.15.72:0/3656963608 <== mon.0 v1:172.21.15.72:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3165667204 0 0) 0x7f86e4131700 con 0x7f86e40b8240 2024-03-29T11:35:02.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86daffd640 1 -- 172.21.15.72:0/3656963608 >> v1:172.21.15.72:6790/0 conn(0x7f86e4141e40 legacy=0x7f86e412cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:02.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86daffd640 1 -- 172.21.15.72:0/3656963608 >> v1:172.21.15.89:6789/0 conn(0x7f86e409c8c0 legacy=0x7f86e4131b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:02.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86daffd640 10 monclient: _finish_hunting 0 2024-03-29T11:35:02.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86daffd640 1 monclient: found mon.a 2024-03-29T11:35:02.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:02.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86daffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:35:02.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86daffd640 1 -- 172.21.15.72:0/3656963608 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86e4134f20 con 0x7f86e40b8240 2024-03-29T11:35:02.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86daffd640 10 monclient: _finish_auth 0 2024-03-29T11:35:02.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86daffd640 10 monclient: _check_auth_tickets 2024-03-29T11:35:02.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:02.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86ec42a640 5 monclient: authenticate success, global_id 4185 2024-03-29T11:35:02.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86ec42a640 10 monclient: _renew_subs 2024-03-29T11:35:02.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86ec42a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:35:02.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86ec42a640 1 -- 172.21.15.72:0/3656963608 --> v1:172.21.15.72:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86e412d3f0 con 0x7f86e40b8240 2024-03-29T11:35:02.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86ec42a640 10 client.4185.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:02.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86ec42a640 10 monclient: _renew_subs 2024-03-29T11:35:02.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86ec42a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:35:02.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86ec42a640 1 -- 172.21.15.72:0/3656963608 --> v1:172.21.15.72:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f86e4144a50 con 0x7f86e40b8240 2024-03-29T11:35:02.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86ec42a640 1 librados: init done 2024-03-29T11:35:02.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86daffd640 1 -- 172.21.15.72:0/3656963608 <== mon.0 v1:172.21.15.72:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86e0003610 con 0x7f86e40b8240 2024-03-29T11:35:02.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86daffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:02.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86daffd640 1 -- 172.21.15.72:0/3656963608 <== mon.0 v1:172.21.15.72:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f86e00039a0 con 0x7f86e40b8240 2024-03-29T11:35:02.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:02.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86daffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:02.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86daffd640 10 monclient: dump: 2024-03-29T11:35:02.516 INFO:teuthology.orchestra.run.smithi072.stderr:epoch 1 2024-03-29T11:35:02.516 INFO:teuthology.orchestra.run.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:02.516 INFO:teuthology.orchestra.run.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:02.516 INFO:teuthology.orchestra.run.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:02.517 INFO:teuthology.orchestra.run.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:02.517 INFO:teuthology.orchestra.run.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:02.517 INFO:teuthology.orchestra.run.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:02.517 INFO:teuthology.orchestra.run.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:02.517 INFO:teuthology.orchestra.run.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:02.517 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-03-29T11:35:02.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86daffd640 1 -- 172.21.15.72:0/3656963608 <== mon.0 v1:172.21.15.72:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f86e0003f50 con 0x7f86e40b8240 2024-03-29T11:35:02.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86e89c3640 4 set_mon_vals no callback set 2024-03-29T11:35:02.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86daffd640 1 -- 172.21.15.72:0/3656963608 <== mon.0 v1:172.21.15.72:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f86e0004200 con 0x7f86e40b8240 2024-03-29T11:35:02.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86daffd640 10 client.4185.objecter ms_dispatch 0x7f86e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:02.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86daffd640 3 client.4185.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:02.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86daffd640 3 client.4185.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:02.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.501+0000 7f86daffd640 20 client.4185.objecter dump_active .. 0 homeless 2024-03-29T11:35:02.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.502+0000 7f86ec42a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:02.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.502+0000 7f86ec42a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:02.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.502+0000 7f86ec42a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:35:02.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.502+0000 7f86ec42a640 1 -- 172.21.15.72:0/3656963608 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86e409b170 con 0x7f86e40b8240 2024-03-29T11:35:02.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.504+0000 7f86daffd640 1 -- 172.21.15.72:0/3656963608 <== mon.0 v1:172.21.15.72:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f86e00057c0 con 0x7f86e40b8240 2024-03-29T11:35:02.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.504+0000 7f86daffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:02.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.504+0000 7f86daffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:02.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.562+0000 7f86ec42a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:02.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.562+0000 7f86ec42a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:02.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.562+0000 7f86ec42a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:35:02.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.562+0000 7f86ec42a640 1 -- 172.21.15.72:0/3656963608 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86e4145250 con 0x7f86e40b8240 2024-03-29T11:35:02.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.563+0000 7f86daffd640 1 -- 172.21.15.72:0/3656963608 <== mon.0 v1:172.21.15.72:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f86e0005a20 con 0x7f86e40b8240 2024-03-29T11:35:02.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.563+0000 7f86daffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:02.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.563+0000 7f86daffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:02.561 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-29T11:35:02.562 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:02.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.564+0000 7f86ec42a640 10 librados: watch_flush enter 2024-03-29T11:35:02.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.564+0000 7f86ec42a640 10 librados: watch_flush exit 2024-03-29T11:35:02.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.564+0000 7f86ec42a640 20 client.4185.objecter shutdown clearing up homeless session... 2024-03-29T11:35:02.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.564+0000 7f86ec42a640 10 client.4185.objecter successfully canceled tick 2024-03-29T11:35:02.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.564+0000 7f86ec42a640 10 monclient: shutdown 2024-03-29T11:35:02.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.564+0000 7f86ec42a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:02.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.564+0000 7f86ec42a640 1 -- 172.21.15.72:0/3656963608 >> v1:172.21.15.72:6789/0 conn(0x7f86e40b8240 legacy=0x7f86e41455e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:02.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.564+0000 7f86ec42a640 1 -- 172.21.15.72:0/3656963608 shutdown_connections 2024-03-29T11:35:02.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.564+0000 7f86ec42a640 1 -- 172.21.15.72:0/3656963608 >> 172.21.15.72:0/3656963608 conn(0x7f86e4066b00 msgr2=0x7f86e4069bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:02.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.564+0000 7f86ec42a640 1 -- 172.21.15.72:0/3656963608 shutdown_connections 2024-03-29T11:35:02.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.564+0000 7f86ec42a640 1 -- 172.21.15.72:0/3656963608 wait complete. 2024-03-29T11:35:02.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-29T11:35:02.564+0000 7f86ec42a640 1 librados: shutdown 2024-03-29T11:35:02.572 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T11:35:02.572 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T11:35:02.585 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-29T11:35:02.587+0000 7fc853e3e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:35:02.600 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-29T11:35:02.602+0000 7f1f7fc2b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:35:02.678 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-29T11:35:02.680+0000 7fc853e3e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:35:02.693 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-29T11:35:02.695+0000 7f1f7fc2b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:35:02.833 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-29T11:35:02.835+0000 7fc853e3e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:35:02.848 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-29T11:35:02.850+0000 7f1f7fc2b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:35:02.875 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-29T11:35:02.877+0000 7fc853e3e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:35:02.889 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-29T11:35:02.891+0000 7f1f7fc2b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:35:02.969 INFO:tasks.ceph.osd.6.smithi089.stderr:2024-03-29T11:35:02.971+0000 7f437ddac640 -1 Falling back to public interface 2024-03-29T11:35:02.975 INFO:tasks.ceph.osd.4.smithi089.stderr:2024-03-29T11:35:02.977+0000 7efcbd136640 -1 Falling back to public interface 2024-03-29T11:35:02.975 INFO:tasks.ceph.osd.1.smithi072.stderr:2024-03-29T11:35:02.977+0000 7fb89a3bd640 -1 Falling back to public interface 2024-03-29T11:35:02.976 INFO:tasks.ceph.osd.0.smithi072.stderr:2024-03-29T11:35:02.978+0000 7ff7a1019640 -1 Falling back to public interface 2024-03-29T11:35:02.983 INFO:tasks.ceph.osd.3.smithi072.stderr:2024-03-29T11:35:02.985+0000 7f97b95dd640 -1 Falling back to public interface 2024-03-29T11:35:02.987 INFO:tasks.ceph.osd.2.smithi072.stderr:2024-03-29T11:35:02.989+0000 7f1d3319e640 -1 Falling back to public interface 2024-03-29T11:35:02.990 INFO:tasks.ceph.osd.5.smithi089.stderr:2024-03-29T11:35:02.992+0000 7fe758664640 -1 Falling back to public interface 2024-03-29T11:35:03.001 INFO:tasks.ceph.osd.7.smithi089.stderr:2024-03-29T11:35:03.003+0000 7ff2a32bd640 -1 Falling back to public interface 2024-03-29T11:35:03.028 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-29T11:35:03.030+0000 7fc853e3e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:35:03.047 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-29T11:35:03.049+0000 7f1f7fc2b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:35:03.377 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:03.467 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.469+0000 7f4374ec9640 10 monclient: get_monmap_and_config 2024-03-29T11:35:03.467 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.469+0000 7f4374ec9640 10 monclient: build_initial_monmap 2024-03-29T11:35:03.467 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.469+0000 7f4374ec9640 10 monclient: monmap: 2024-03-29T11:35:03.468 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:35:03.468 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:03.468 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:35:03.470694+0000 2024-03-29T11:35:03.468 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:35:03.470694+0000 2024-03-29T11:35:03.468 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:03.468 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:03.468 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:35:03.468 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:35:03.468 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:35:03.469 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:03.483 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.470+0000 7f4374ec9640 1 Processor -- start 2024-03-29T11:35:03.483 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.470+0000 7f4374ec9640 1 -- start start 2024-03-29T11:35:03.483 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.470+0000 7f4374ec9640 10 monclient: init 2024-03-29T11:35:03.483 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f4374ec9640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:03.483 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f4374ec9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:35:03.483 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f4374ec9640 10 monclient(hunting): picked mon.noname-a con 0x7f43701422b0 addr v1:172.21.15.72:6789/0 2024-03-29T11:35:03.483 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f4374ec9640 10 monclient(hunting): picked mon.noname-b con 0x7f4370142690 addr v1:172.21.15.72:6790/0 2024-03-29T11:35:03.484 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f4374ec9640 10 monclient(hunting): picked mon.noname-c con 0x7f43700b17d0 addr v1:172.21.15.89:6789/0 2024-03-29T11:35:03.484 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f4374ec9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4370142c80 con 0x7f43701422b0 2024-03-29T11:35:03.484 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f4374ec9640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43700aae40 con 0x7f43700b17d0 2024-03-29T11:35:03.484 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f4374ec9640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f437013fbd0 con 0x7f4370142690 2024-03-29T11:35:03.484 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f4374ec9640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:03.484 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f4374ec9640 10 monclient(hunting): authenticate will time out at 862.759081s 2024-03-29T11:35:03.484 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f436e59c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f43701422b0 0x7f437013f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:53714/0 (socket says 172.21.15.72:53714) 2024-03-29T11:35:03.484 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f436e59c640 1 -- 172.21.15.72:0/999155591 learned_addr learned my addr 172.21.15.72:0/999155591 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:35:03.484 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f436dd9b640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f4370142690 0x7f43700b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:37692/0 (socket says 172.21.15.72:37692) 2024-03-29T11:35:03.484 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f436d59a640 1 -- 172.21.15.72:0/999155591 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1235891323 0 0) 0x7f4370142c80 con 0x7f43701422b0 2024-03-29T11:35:03.484 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f436d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:35:03.484 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f436d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:03.484 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f436d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:03.484 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f436d59a640 10 monclient(hunting): my global_id is 4188 2024-03-29T11:35:03.484 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f436d59a640 1 -- 172.21.15.72:0/999155591 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4358003520 con 0x7f43701422b0 2024-03-29T11:35:03.485 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f436d59a640 1 -- 172.21.15.72:0/999155591 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3163951508 0 0) 0x7f437013fbd0 con 0x7f4370142690 2024-03-29T11:35:03.485 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f436d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:35:03.485 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f436d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:03.485 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f436d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:03.485 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f436d59a640 10 monclient(hunting): my global_id is 4181 2024-03-29T11:35:03.485 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f436d59a640 1 -- 172.21.15.72:0/999155591 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4370142c80 con 0x7f4370142690 2024-03-29T11:35:03.485 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f436d59a640 1 -- 172.21.15.72:0/999155591 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1017066066 0 0) 0x7f43700aae40 con 0x7f43700b17d0 2024-03-29T11:35:03.485 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f436d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:35:03.485 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f436d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:03.485 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f436d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:03.485 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f436d59a640 10 monclient(hunting): my global_id is 4192 2024-03-29T11:35:03.485 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f436d59a640 1 -- 172.21.15.72:0/999155591 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f437013fbd0 con 0x7f43700b17d0 2024-03-29T11:35:03.485 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f436d59a640 1 -- 172.21.15.72:0/999155591 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2060049088 0 0) 0x7f4358003520 con 0x7f43701422b0 2024-03-29T11:35:03.485 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f436d59a640 1 -- 172.21.15.72:0/999155591 >> v1:172.21.15.72:6790/0 conn(0x7f4370142690 legacy=0x7f43700b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:03.485 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f436d59a640 1 -- 172.21.15.72:0/999155591 >> v1:172.21.15.89:6789/0 conn(0x7f43700b17d0 legacy=0x7f43700b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:03.486 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f436d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:35:03.486 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f436d59a640 1 monclient: found mon.noname-a 2024-03-29T11:35:03.486 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f436d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:03.486 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f436d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.72:6789/0 2024-03-29T11:35:03.486 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f436d59a640 1 -- 172.21.15.72:0/999155591 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4370151940 con 0x7f43701422b0 2024-03-29T11:35:03.486 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f436d59a640 10 monclient: _finish_auth 0 2024-03-29T11:35:03.486 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f436d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:35:03.486 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f436d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:03.486 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f4374ec9640 5 monclient: authenticate success, global_id 4188 2024-03-29T11:35:03.486 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f436d59a640 1 -- 172.21.15.72:0/999155591 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f4360001690 con 0x7f43701422b0 2024-03-29T11:35:03.486 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f436d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:03.486 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f436d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:35:03.486 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f436d59a640 10 monclient: dump: 2024-03-29T11:35:03.486 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:35:03.486 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:03.487 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:03.487 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:03.487 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:03.487 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:03.487 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:03.487 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:03.487 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:03.487 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:03.487 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f4374ec9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:03.487 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f436d59a640 1 -- 172.21.15.72:0/999155591 <== mon.0 v1:172.21.15.72:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4360001870 con 0x7f43701422b0 2024-03-29T11:35:03.487 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f436d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:03.487 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f436d59a640 1 -- 172.21.15.72:0/999155591 <== mon.0 v1:172.21.15.72:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f4360001c40 con 0x7f43701422b0 2024-03-29T11:35:03.487 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f4374ec9640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:03.487 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f4374ec9640 4 set_mon_vals no callback set 2024-03-29T11:35:03.487 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f436d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:03.487 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f436d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:03.488 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.471+0000 7f436d59a640 10 monclient: dump: 2024-03-29T11:35:03.488 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:35:03.488 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:03.488 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:03.488 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:03.488 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:03.488 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:03.488 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:03.488 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:03.488 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:03.488 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:03.488 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.472+0000 7f4374ec9640 10 monclient: shutdown 2024-03-29T11:35:03.489 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.472+0000 7f4374ec9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:03.489 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.472+0000 7f4374ec9640 1 -- 172.21.15.72:0/999155591 >> v1:172.21.15.72:6789/0 conn(0x7f43701422b0 legacy=0x7f437013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:03.489 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.472+0000 7f4374ec9640 1 -- 172.21.15.72:0/999155591 shutdown_connections 2024-03-29T11:35:03.489 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.472+0000 7f4374ec9640 1 -- 172.21.15.72:0/999155591 >> 172.21.15.72:0/999155591 conn(0x7f4370066a00 msgr2=0x7f4370066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:03.489 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.472+0000 7f4374ec9640 1 -- 172.21.15.72:0/999155591 shutdown_connections 2024-03-29T11:35:03.489 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.472+0000 7f4374ec9640 1 -- 172.21.15.72:0/999155591 wait complete. 2024-03-29T11:35:03.489 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.473+0000 7f4374ec9640 10 monclient: build_initial_monmap 2024-03-29T11:35:03.489 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.473+0000 7f4374ec9640 10 monclient: monmap: 2024-03-29T11:35:03.489 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:35:03.489 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:03.489 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:35:03.489 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:35:03.489 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:03.489 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:03.490 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:35:03.490 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:35:03.490 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:35:03.490 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:03.490 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.473+0000 7f4374ec9640 1 librados: starting msgr at 2024-03-29T11:35:03.490 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.473+0000 7f4374ec9640 1 librados: starting objecter 2024-03-29T11:35:03.490 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.473+0000 7f4374ec9640 1 Processor -- start 2024-03-29T11:35:03.490 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.473+0000 7f4374ec9640 1 -- start start 2024-03-29T11:35:03.490 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.473+0000 7f4374ec9640 1 librados: setting wanted keys 2024-03-29T11:35:03.490 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.473+0000 7f4374ec9640 1 librados: calling monclient init 2024-03-29T11:35:03.490 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.473+0000 7f4374ec9640 10 monclient: init 2024-03-29T11:35:03.490 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.473+0000 7f4374ec9640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:03.490 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.473+0000 7f4374ec9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:35:03.490 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f4374ec9640 10 monclient(hunting): picked mon.noname-b con 0x7f43700b17d0 addr v1:172.21.15.89:6789/0 2024-03-29T11:35:03.490 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f4374ec9640 10 monclient(hunting): picked mon.noname-c con 0x7f43701422b0 addr v1:172.21.15.72:6790/0 2024-03-29T11:35:03.491 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f4374ec9640 10 monclient(hunting): picked mon.noname-a con 0x7f4370142690 addr v1:172.21.15.72:6789/0 2024-03-29T11:35:03.491 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f4374ec9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f437014f960 con 0x7f4370142690 2024-03-29T11:35:03.491 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f4374ec9640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43700acb30 con 0x7f43700b17d0 2024-03-29T11:35:03.491 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f4374ec9640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43700ace10 con 0x7f43701422b0 2024-03-29T11:35:03.491 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f4374ec9640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:03.491 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f4374ec9640 10 monclient(hunting): authenticate will time out at 862.762063s 2024-03-29T11:35:03.491 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f436ed9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f4370142690 0x7f43701e2a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:53720/0 (socket says 172.21.15.72:53720) 2024-03-29T11:35:03.491 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f436ed9d640 1 -- 172.21.15.72:0/1609137844 learned_addr learned my addr 172.21.15.72:0/1609137844 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:35:03.491 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f436dd9b640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f43701422b0 0x7f437012cfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:37702/0 (socket says 172.21.15.72:37702) 2024-03-29T11:35:03.491 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 10 client.?.objecter ms_handle_connect 0x7f43701422b0 2024-03-29T11:35:03.491 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:03.491 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 10 client.?.objecter ms_handle_connect 0x7f4370142690 2024-03-29T11:35:03.491 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:03.491 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 10 client.?.objecter ms_handle_connect 0x7f43700b17d0 2024-03-29T11:35:03.491 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:03.492 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 1 -- 172.21.15.72:0/1609137844 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 450851049 0 0) 0x7f437014f960 con 0x7f4370142690 2024-03-29T11:35:03.492 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:03.492 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:03.492 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:03.492 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 10 monclient(hunting): my global_id is 4191 2024-03-29T11:35:03.492 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 1 -- 172.21.15.72:0/1609137844 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4350004eb0 con 0x7f4370142690 2024-03-29T11:35:03.492 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 1 -- 172.21.15.72:0/1609137844 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3130729288 0 0) 0x7f43700ace10 con 0x7f43701422b0 2024-03-29T11:35:03.492 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:03.492 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:03.492 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:03.492 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 10 monclient(hunting): my global_id is 4184 2024-03-29T11:35:03.492 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 1 -- 172.21.15.72:0/1609137844 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f437014f960 con 0x7f43701422b0 2024-03-29T11:35:03.492 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 1 -- 172.21.15.72:0/1609137844 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2668434999 0 0) 0x7f43700acb30 con 0x7f43700b17d0 2024-03-29T11:35:03.492 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:03.492 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:03.492 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:03.493 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 10 monclient(hunting): my global_id is 4195 2024-03-29T11:35:03.493 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 1 -- 172.21.15.72:0/1609137844 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43700ace10 con 0x7f43700b17d0 2024-03-29T11:35:03.493 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 1 -- 172.21.15.72:0/1609137844 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3376389356 0 0) 0x7f437014f960 con 0x7f43701422b0 2024-03-29T11:35:03.493 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 1 -- 172.21.15.72:0/1609137844 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43700acb30 con 0x7f43701422b0 2024-03-29T11:35:03.493 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 1 -- 172.21.15.72:0/1609137844 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f43640017f0 con 0x7f43701422b0 2024-03-29T11:35:03.493 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:03.493 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:03.493 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 10 monclient(hunting): dump: 2024-03-29T11:35:03.493 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:35:03.493 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:03.493 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:03.493 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:03.493 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:03.493 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:03.493 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:03.493 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:03.494 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:03.494 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:03.494 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:03.494 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 1 -- 172.21.15.72:0/1609137844 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2255976543 0 0) 0x7f4350004eb0 con 0x7f4370142690 2024-03-29T11:35:03.494 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 1 -- 172.21.15.72:0/1609137844 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f437014f960 con 0x7f4370142690 2024-03-29T11:35:03.494 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 1 -- 172.21.15.72:0/1609137844 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 688569460 0 0) 0x7f43700ace10 con 0x7f43700b17d0 2024-03-29T11:35:03.494 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 1 -- 172.21.15.72:0/1609137844 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4350004eb0 con 0x7f43700b17d0 2024-03-29T11:35:03.494 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 1 -- 172.21.15.72:0/1609137844 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f4368001d10 con 0x7f4370142690 2024-03-29T11:35:03.494 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:03.494 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:03.494 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 10 monclient(hunting): dump: 2024-03-29T11:35:03.494 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:35:03.494 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:03.494 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:03.495 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:03.495 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:03.495 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:03.495 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:03.495 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:03.495 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:03.495 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:03.495 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 1 -- 172.21.15.72:0/1609137844 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f43600019e0 con 0x7f43700b17d0 2024-03-29T11:35:03.495 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:03.495 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:03.495 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 10 monclient(hunting): dump: 2024-03-29T11:35:03.495 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:35:03.495 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:03.495 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:03.495 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:03.495 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:03.496 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:03.496 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:03.496 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:03.496 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:03.496 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:03.496 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 1 -- 172.21.15.72:0/1609137844 <== mon.2 v1:172.21.15.72:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2546252645 0 0) 0x7f43700acb30 con 0x7f43701422b0 2024-03-29T11:35:03.496 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 1 -- 172.21.15.72:0/1609137844 >> v1:172.21.15.89:6789/0 conn(0x7f43700b17d0 legacy=0x7f437014f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:03.496 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 1 -- 172.21.15.72:0/1609137844 >> v1:172.21.15.72:6789/0 conn(0x7f4370142690 legacy=0x7f43701e2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:03.496 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 10 monclient: _finish_hunting 0 2024-03-29T11:35:03.496 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 1 monclient: found mon.c 2024-03-29T11:35:03.496 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:03.496 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:35:03.496 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 1 -- 172.21.15.72:0/1609137844 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4370150c50 con 0x7f43701422b0 2024-03-29T11:35:03.496 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 10 monclient: _finish_auth 0 2024-03-29T11:35:03.496 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 10 monclient: _check_auth_tickets 2024-03-29T11:35:03.497 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f435effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:03.497 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f4374ec9640 5 monclient: authenticate success, global_id 4184 2024-03-29T11:35:03.497 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f4374ec9640 10 monclient: _renew_subs 2024-03-29T11:35:03.497 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f4374ec9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:35:03.497 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.474+0000 7f4374ec9640 1 -- 172.21.15.72:0/1609137844 --> v1:172.21.15.72:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4370151180 con 0x7f43701422b0 2024-03-29T11:35:03.497 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.475+0000 7f4374ec9640 10 client.4184.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:03.497 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.475+0000 7f4374ec9640 10 monclient: _renew_subs 2024-03-29T11:35:03.497 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.475+0000 7f4374ec9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:35:03.497 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.475+0000 7f4374ec9640 1 -- 172.21.15.72:0/1609137844 --> v1:172.21.15.72:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f43700b6710 con 0x7f43701422b0 2024-03-29T11:35:03.497 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.475+0000 7f4374ec9640 1 librados: init done 2024-03-29T11:35:03.497 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.475+0000 7f435effd640 1 -- 172.21.15.72:0/1609137844 <== mon.2 v1:172.21.15.72:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4364001ca0 con 0x7f43701422b0 2024-03-29T11:35:03.497 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.475+0000 7f435effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:03.497 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.475+0000 7f435effd640 1 -- 172.21.15.72:0/1609137844 <== mon.2 v1:172.21.15.72:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f4364004980 con 0x7f43701422b0 2024-03-29T11:35:03.497 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.475+0000 7f435effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:03.497 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.475+0000 7f435effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:03.498 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.475+0000 7f435effd640 10 monclient: dump: 2024-03-29T11:35:03.498 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:35:03.498 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:03.498 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:03.498 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:03.498 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:03.498 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:03.498 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:03.498 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:03.498 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:03.498 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:03.498 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.475+0000 7f436cd99640 4 set_mon_vals no callback set 2024-03-29T11:35:03.498 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.475+0000 7f435effd640 1 -- 172.21.15.72:0/1609137844 <== mon.2 v1:172.21.15.72:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f4364004eb0 con 0x7f43701422b0 2024-03-29T11:35:03.498 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.475+0000 7f435effd640 1 -- 172.21.15.72:0/1609137844 <== mon.2 v1:172.21.15.72:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f4364006790 con 0x7f43701422b0 2024-03-29T11:35:03.498 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.475+0000 7f435effd640 10 client.4184.objecter ms_dispatch 0x7f4370000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:03.499 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.475+0000 7f435effd640 3 client.4184.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:03.499 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.475+0000 7f435effd640 3 client.4184.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:03.499 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.475+0000 7f435effd640 20 client.4184.objecter dump_active .. 0 homeless 2024-03-29T11:35:03.499 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.476+0000 7f4374ec9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:03.499 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.476+0000 7f4374ec9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:03.499 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.476+0000 7f4374ec9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:35:03.499 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.476+0000 7f4374ec9640 1 -- 172.21.15.72:0/1609137844 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4370150e40 con 0x7f43701422b0 2024-03-29T11:35:03.499 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.478+0000 7f435effd640 1 -- 172.21.15.72:0/1609137844 <== mon.2 v1:172.21.15.72:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f4364006a20 con 0x7f43701422b0 2024-03-29T11:35:03.499 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.478+0000 7f435effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:03.499 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.478+0000 7f435effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:03.537 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.539+0000 7f4374ec9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:03.537 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.539+0000 7f4374ec9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:03.537 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.539+0000 7f4374ec9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:35:03.537 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.539+0000 7f4374ec9640 1 -- 172.21.15.72:0/1609137844 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4370075f10 con 0x7f43701422b0 2024-03-29T11:35:03.538 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.540+0000 7f435effd640 1 -- 172.21.15.72:0/1609137844 <== mon.2 v1:172.21.15.72:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f4364006280 con 0x7f43701422b0 2024-03-29T11:35:03.538 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.540+0000 7f435effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:03.538 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.540+0000 7f435effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:03.538 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:35:03.538 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:03.539 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.540+0000 7f4374ec9640 10 librados: watch_flush enter 2024-03-29T11:35:03.540 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.540+0000 7f4374ec9640 10 librados: watch_flush exit 2024-03-29T11:35:03.540 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.541+0000 7f4374ec9640 20 client.4184.objecter shutdown clearing up homeless session... 2024-03-29T11:35:03.540 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.541+0000 7f4374ec9640 10 client.4184.objecter successfully canceled tick 2024-03-29T11:35:03.540 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.541+0000 7f4374ec9640 10 monclient: shutdown 2024-03-29T11:35:03.540 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.541+0000 7f4374ec9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:03.540 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.541+0000 7f4374ec9640 1 -- 172.21.15.72:0/1609137844 >> v1:172.21.15.72:6790/0 conn(0x7f43701422b0 legacy=0x7f437012cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:03.540 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.541+0000 7f4374ec9640 1 -- 172.21.15.72:0/1609137844 shutdown_connections 2024-03-29T11:35:03.540 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.541+0000 7f4374ec9640 1 -- 172.21.15.72:0/1609137844 >> 172.21.15.72:0/1609137844 conn(0x7f4370066a00 msgr2=0x7f437013c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:03.540 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.541+0000 7f4374ec9640 1 -- 172.21.15.72:0/1609137844 shutdown_connections 2024-03-29T11:35:03.540 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.541+0000 7f4374ec9640 1 -- 172.21.15.72:0/1609137844 wait complete. 2024-03-29T11:35:03.540 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:03.541+0000 7f4374ec9640 1 librados: shutdown 2024-03-29T11:35:03.547 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:03.871 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-29T11:35:03.873+0000 7fc853e3e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:35:03.880 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-29T11:35:03.882+0000 7f1f7fc2b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:35:04.142 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-29T11:35:04.145+0000 7fc853e3e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:35:04.155 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-29T11:35:04.157+0000 7f1f7fc2b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:35:04.193 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-29T11:35:04.195+0000 7fc853e3e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:35:04.205 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-29T11:35:04.207+0000 7f1f7fc2b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:35:04.305 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-29T11:35:04.307+0000 7fc853e3e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:35:04.321 INFO:tasks.ceph.mgr.x.smithi089.stderr:2024-03-29T11:35:04.323+0000 7f1f7fc2b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:35:05.319 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-29T11:35:05.321+0000 7fc7f4a3d640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T11:35:05.320 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-03-29T11:35:05.322+0000 7fc7f4a3d640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T11:35:06.434 INFO:tasks.ceph.osd.6.smithi089.stderr:2024-03-29T11:35:06.436+0000 7f437ddac640 -1 osd.6 0 log_to_monitors true 2024-03-29T11:35:06.486 INFO:tasks.ceph.osd.0.smithi072.stderr:2024-03-29T11:35:06.488+0000 7ff7a1019640 -1 osd.0 0 log_to_monitors true 2024-03-29T11:35:06.492 INFO:tasks.ceph.osd.4.smithi089.stderr:2024-03-29T11:35:06.494+0000 7efcbd136640 -1 osd.4 0 log_to_monitors true 2024-03-29T11:35:06.492 INFO:tasks.ceph.osd.5.smithi089.stderr:2024-03-29T11:35:06.494+0000 7fe758664640 -1 osd.5 0 log_to_monitors true 2024-03-29T11:35:06.494 INFO:tasks.ceph.osd.1.smithi072.stderr:2024-03-29T11:35:06.496+0000 7fb89a3bd640 -1 osd.1 0 log_to_monitors true 2024-03-29T11:35:06.498 INFO:tasks.ceph.osd.3.smithi072.stderr:2024-03-29T11:35:06.500+0000 7f97b95dd640 -1 osd.3 0 log_to_monitors true 2024-03-29T11:35:06.523 INFO:tasks.ceph.osd.7.smithi089.stderr:2024-03-29T11:35:06.525+0000 7ff2a32bd640 -1 osd.7 0 log_to_monitors true 2024-03-29T11:35:06.527 INFO:tasks.ceph.osd.2.smithi072.stderr:2024-03-29T11:35:06.529+0000 7f1d3319e640 -1 osd.2 0 log_to_monitors true 2024-03-29T11:35:10.356 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:10.502 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.503+0000 7f35555af640 10 monclient: get_monmap_and_config 2024-03-29T11:35:10.502 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.503+0000 7f35555af640 10 monclient: build_initial_monmap 2024-03-29T11:35:10.503 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.503+0000 7f35555af640 10 monclient: monmap: 2024-03-29T11:35:10.503 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:35:10.503 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:10.503 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:35:10.504882+0000 2024-03-29T11:35:10.503 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:35:10.504882+0000 2024-03-29T11:35:10.503 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:10.503 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:10.504 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:35:10.504 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:35:10.504 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:35:10.504 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:10.521 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.505+0000 7f35555af640 1 Processor -- start 2024-03-29T11:35:10.521 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.505+0000 7f35555af640 1 -- start start 2024-03-29T11:35:10.522 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.505+0000 7f35555af640 10 monclient: init 2024-03-29T11:35:10.522 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.506+0000 7f35555af640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:10.522 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.506+0000 7f35555af640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:35:10.522 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.506+0000 7f35555af640 10 monclient(hunting): picked mon.noname-b con 0x7f355006cb30 addr v1:172.21.15.72:6790/0 2024-03-29T11:35:10.522 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.506+0000 7f35555af640 10 monclient(hunting): picked mon.noname-c con 0x7f355006cf10 addr v1:172.21.15.89:6789/0 2024-03-29T11:35:10.522 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.507+0000 7f354effd640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f355006cb30 0x7f3550071f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:46060/0 (socket says 172.21.15.72:46060) 2024-03-29T11:35:10.523 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.507+0000 7f354effd640 1 -- 172.21.15.72:0/3233205635 learned_addr learned my addr 172.21.15.72:0/3233205635 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:35:10.523 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.507+0000 7f35555af640 10 monclient(hunting): picked mon.noname-a con 0x7f3550142960 addr v1:172.21.15.72:6789/0 2024-03-29T11:35:10.523 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.507+0000 7f35555af640 1 -- 172.21.15.72:0/3233205635 --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3550142dd0 con 0x7f3550142960 2024-03-29T11:35:10.523 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.507+0000 7f35555af640 1 -- 172.21.15.72:0/3233205635 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35500acc40 con 0x7f355006cf10 2024-03-29T11:35:10.523 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.507+0000 7f35555af640 1 -- 172.21.15.72:0/3233205635 --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3550072620 con 0x7f355006cb30 2024-03-29T11:35:10.523 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.507+0000 7f35555af640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:10.523 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.507+0000 7f35555af640 10 monclient(hunting): authenticate will time out at 869.795361s 2024-03-29T11:35:10.523 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.507+0000 7f354e7fc640 1 -- 172.21.15.72:0/3233205635 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1831366345 0 0) 0x7f3550072620 con 0x7f355006cb30 2024-03-29T11:35:10.524 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.507+0000 7f354e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:10.524 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.507+0000 7f354e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:10.524 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.507+0000 7f354e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:10.524 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.507+0000 7f354e7fc640 10 monclient(hunting): my global_id is 4229 2024-03-29T11:35:10.524 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.507+0000 7f354e7fc640 1 -- 172.21.15.72:0/3233205635 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f353c003520 con 0x7f355006cb30 2024-03-29T11:35:10.524 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.507+0000 7f354e7fc640 1 -- 172.21.15.72:0/3233205635 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1312982027 0 0) 0x7f35500acc40 con 0x7f355006cf10 2024-03-29T11:35:10.524 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.507+0000 7f354e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:10.525 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.507+0000 7f354e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:10.525 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.507+0000 7f354e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:10.525 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.507+0000 7f354e7fc640 10 monclient(hunting): my global_id is 4240 2024-03-29T11:35:10.525 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.507+0000 7f354e7fc640 1 -- 172.21.15.72:0/3233205635 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3550072620 con 0x7f355006cf10 2024-03-29T11:35:10.525 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.508+0000 7f354e7fc640 1 -- 172.21.15.72:0/3233205635 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2130171892 0 0) 0x7f3550072620 con 0x7f355006cf10 2024-03-29T11:35:10.525 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.508+0000 7f354e7fc640 1 -- 172.21.15.72:0/3233205635 >> v1:172.21.15.72:6790/0 conn(0x7f355006cb30 legacy=0x7f3550071f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:10.525 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.508+0000 7f354e7fc640 1 -- 172.21.15.72:0/3233205635 >> v1:172.21.15.72:6789/0 conn(0x7f3550142960 legacy=0x7f355014da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:10.526 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.508+0000 7f354e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:35:10.526 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.508+0000 7f354e7fc640 1 monclient: found mon.noname-c 2024-03-29T11:35:10.526 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.508+0000 7f354e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:10.526 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.508+0000 7f354e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.89:6789/0 2024-03-29T11:35:10.526 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.508+0000 7f354e7fc640 1 -- 172.21.15.72:0/3233205635 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3550070190 con 0x7f355006cf10 2024-03-29T11:35:10.526 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.508+0000 7f354e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:35:10.527 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.508+0000 7f354e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:35:10.527 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.508+0000 7f354e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:10.527 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.508+0000 7f354e7fc640 1 -- 172.21.15.72:0/3233205635 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f3538001670 con 0x7f355006cf10 2024-03-29T11:35:10.527 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.508+0000 7f35555af640 5 monclient: authenticate success, global_id 4240 2024-03-29T11:35:10.527 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.508+0000 7f35555af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:10.527 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.508+0000 7f354e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:10.527 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.508+0000 7f354e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:10.528 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.508+0000 7f354e7fc640 10 monclient: dump: 2024-03-29T11:35:10.528 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:35:10.528 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:10.528 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:10.528 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:10.528 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:10.528 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:10.529 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:10.529 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:10.529 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:10.529 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:10.529 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.508+0000 7f35555af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:10.529 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.508+0000 7f354e7fc640 1 -- 172.21.15.72:0/3233205635 <== mon.1 v1:172.21.15.89:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35380017d0 con 0x7f355006cf10 2024-03-29T11:35:10.529 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.508+0000 7f354e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:10.530 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.508+0000 7f354e7fc640 1 -- 172.21.15.72:0/3233205635 <== mon.1 v1:172.21.15.89:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f3538001ba0 con 0x7f355006cf10 2024-03-29T11:35:10.530 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.508+0000 7f354e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:10.530 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.508+0000 7f354e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:10.530 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.508+0000 7f354e7fc640 10 monclient: dump: 2024-03-29T11:35:10.530 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:35:10.530 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:10.531 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:10.531 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:10.531 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:10.531 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:10.531 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:10.531 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:10.531 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:10.532 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:10.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.508+0000 7f35555af640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:10.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.508+0000 7f35555af640 4 set_mon_vals no callback set 2024-03-29T11:35:10.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.509+0000 7f35555af640 10 monclient: shutdown 2024-03-29T11:35:10.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.509+0000 7f35555af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:10.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.509+0000 7f35555af640 1 -- 172.21.15.72:0/3233205635 >> v1:172.21.15.89:6789/0 conn(0x7f355006cf10 legacy=0x7f3550142250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:10.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.510+0000 7f35555af640 1 -- 172.21.15.72:0/3233205635 shutdown_connections 2024-03-29T11:35:10.533 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.510+0000 7f35555af640 1 -- 172.21.15.72:0/3233205635 >> 172.21.15.72:0/3233205635 conn(0x7f35500b5a20 msgr2=0x7f35500b5e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:10.533 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.510+0000 7f35555af640 1 -- 172.21.15.72:0/3233205635 shutdown_connections 2024-03-29T11:35:10.533 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.510+0000 7f35555af640 1 -- 172.21.15.72:0/3233205635 wait complete. 2024-03-29T11:35:10.533 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.511+0000 7f35555af640 10 monclient: build_initial_monmap 2024-03-29T11:35:10.533 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.511+0000 7f35555af640 10 monclient: monmap: 2024-03-29T11:35:10.533 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:35:10.533 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:10.534 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:35:10.534 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:35:10.534 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:10.534 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:10.534 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:35:10.534 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:35:10.535 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:35:10.535 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:10.535 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.512+0000 7f35555af640 1 librados: starting msgr at 2024-03-29T11:35:10.535 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.512+0000 7f35555af640 1 librados: starting objecter 2024-03-29T11:35:10.535 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.512+0000 7f35555af640 1 Processor -- start 2024-03-29T11:35:10.536 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.512+0000 7f35555af640 1 -- start start 2024-03-29T11:35:10.536 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.512+0000 7f35555af640 1 librados: setting wanted keys 2024-03-29T11:35:10.536 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.512+0000 7f35555af640 1 librados: calling monclient init 2024-03-29T11:35:10.536 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.512+0000 7f35555af640 10 monclient: init 2024-03-29T11:35:10.536 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.513+0000 7f35555af640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:10.537 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.513+0000 7f35555af640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:35:10.537 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.513+0000 7f35555af640 10 monclient(hunting): picked mon.noname-b con 0x7f355006cb30 addr v1:172.21.15.89:6789/0 2024-03-29T11:35:10.537 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.513+0000 7f35555af640 10 monclient(hunting): picked mon.noname-c con 0x7f355006cf10 addr v1:172.21.15.72:6790/0 2024-03-29T11:35:10.537 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.513+0000 7f35555af640 10 monclient(hunting): picked mon.noname-a con 0x7f3550142960 addr v1:172.21.15.72:6789/0 2024-03-29T11:35:10.537 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.513+0000 7f35555af640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35501d7660 con 0x7f3550142960 2024-03-29T11:35:10.538 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.513+0000 7f35555af640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f355013ddc0 con 0x7f355006cb30 2024-03-29T11:35:10.538 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.513+0000 7f35555af640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f355013b9d0 con 0x7f355006cf10 2024-03-29T11:35:10.538 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.513+0000 7f35555af640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:10.538 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.513+0000 7f35555af640 10 monclient(hunting): authenticate will time out at 869.801606s 2024-03-29T11:35:10.538 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.513+0000 7f354effd640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f355006cb30 0x7f35501d7af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:36218/0 (socket says 172.21.15.72:36218) 2024-03-29T11:35:10.539 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.513+0000 7f354effd640 1 -- 172.21.15.72:0/2626140941 learned_addr learned my addr 172.21.15.72:0/2626140941 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:35:10.539 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.514+0000 7f35477fe640 10 client.?.objecter ms_handle_connect 0x7f355006cb30 2024-03-29T11:35:10.539 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.514+0000 7f35477fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:10.539 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.514+0000 7f35477fe640 10 client.?.objecter ms_handle_connect 0x7f355006cf10 2024-03-29T11:35:10.539 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.514+0000 7f35477fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:10.540 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.514+0000 7f35477fe640 1 -- 172.21.15.72:0/2626140941 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3193488406 0 0) 0x7f355013ddc0 con 0x7f355006cb30 2024-03-29T11:35:10.540 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.514+0000 7f35477fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:10.540 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.514+0000 7f35477fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:10.540 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.514+0000 7f35477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:10.540 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.514+0000 7f35477fe640 10 monclient(hunting): my global_id is 4243 2024-03-29T11:35:10.541 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.514+0000 7f35477fe640 1 -- 172.21.15.72:0/2626140941 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3530004eb0 con 0x7f355006cb30 2024-03-29T11:35:10.541 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.514+0000 7f35477fe640 1 -- 172.21.15.72:0/2626140941 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1831229491 0 0) 0x7f3530004eb0 con 0x7f355006cb30 2024-03-29T11:35:10.541 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.514+0000 7f35477fe640 1 -- 172.21.15.72:0/2626140941 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f355013ddc0 con 0x7f355006cb30 2024-03-29T11:35:10.541 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.514+0000 7f35477fe640 1 -- 172.21.15.72:0/2626140941 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f3540001950 con 0x7f355006cb30 2024-03-29T11:35:10.541 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.514+0000 7f35477fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:10.542 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.515+0000 7f35477fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:10.542 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.515+0000 7f35477fe640 10 monclient(hunting): dump: 2024-03-29T11:35:10.542 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:35:10.542 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:10.543 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:10.543 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:10.543 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:10.543 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:10.543 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:10.544 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:10.544 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:10.544 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:10.544 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.515+0000 7f35477fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:10.544 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.515+0000 7f35477fe640 10 client.?.objecter ms_handle_connect 0x7f3550142960 2024-03-29T11:35:10.544 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.515+0000 7f35477fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:10.545 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.515+0000 7f35477fe640 1 -- 172.21.15.72:0/2626140941 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 934550392 0 0) 0x7f355013b9d0 con 0x7f355006cf10 2024-03-29T11:35:10.545 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.515+0000 7f35477fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:10.545 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.515+0000 7f35477fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:10.545 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.515+0000 7f35477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:10.545 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.515+0000 7f35477fe640 10 monclient(hunting): my global_id is 4232 2024-03-29T11:35:10.546 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.515+0000 7f35477fe640 1 -- 172.21.15.72:0/2626140941 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3530004eb0 con 0x7f355006cf10 2024-03-29T11:35:10.546 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.515+0000 7f35477fe640 1 -- 172.21.15.72:0/2626140941 <== mon.1 v1:172.21.15.89:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1675784466 0 0) 0x7f355013ddc0 con 0x7f355006cb30 2024-03-29T11:35:10.546 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.515+0000 7f35477fe640 1 -- 172.21.15.72:0/2626140941 >> v1:172.21.15.72:6790/0 conn(0x7f355006cf10 legacy=0x7f35501d45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:10.546 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.515+0000 7f35477fe640 1 -- 172.21.15.72:0/2626140941 >> v1:172.21.15.72:6789/0 conn(0x7f3550142960 legacy=0x7f355013c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:10.547 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.515+0000 7f35477fe640 10 monclient: _finish_hunting 0 2024-03-29T11:35:10.547 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.515+0000 7f35477fe640 1 monclient: found mon.b 2024-03-29T11:35:10.547 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.515+0000 7f35477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:10.547 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.515+0000 7f35477fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:10.547 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.515+0000 7f35477fe640 1 -- 172.21.15.72:0/2626140941 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35501d8200 con 0x7f355006cb30 2024-03-29T11:35:10.548 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.515+0000 7f35477fe640 10 monclient: _finish_auth 0 2024-03-29T11:35:10.548 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.515+0000 7f35477fe640 10 monclient: _check_auth_tickets 2024-03-29T11:35:10.548 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.515+0000 7f35477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:10.548 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.515+0000 7f35555af640 5 monclient: authenticate success, global_id 4243 2024-03-29T11:35:10.548 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.515+0000 7f35555af640 10 monclient: _renew_subs 2024-03-29T11:35:10.548 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.515+0000 7f35555af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:10.549 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.515+0000 7f35555af640 1 -- 172.21.15.72:0/2626140941 --> v1:172.21.15.89:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35501d3f50 con 0x7f355006cb30 2024-03-29T11:35:10.549 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.515+0000 7f35555af640 10 client.4243.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:10.549 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.515+0000 7f35555af640 10 monclient: _renew_subs 2024-03-29T11:35:10.549 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.515+0000 7f35555af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:10.550 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.515+0000 7f35555af640 1 -- 172.21.15.72:0/2626140941 --> v1:172.21.15.89:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f35501d4ec0 con 0x7f355006cb30 2024-03-29T11:35:10.550 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.515+0000 7f35555af640 1 librados: init done 2024-03-29T11:35:10.550 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.516+0000 7f35477fe640 1 -- 172.21.15.72:0/2626140941 <== mon.1 v1:172.21.15.89:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35400047e0 con 0x7f355006cb30 2024-03-29T11:35:10.550 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.516+0000 7f35477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:10.550 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.516+0000 7f354dffb640 4 set_mon_vals no callback set 2024-03-29T11:35:10.551 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.517+0000 7f35477fe640 1 -- 172.21.15.72:0/2626140941 <== mon.1 v1:172.21.15.89:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f3540004bf0 con 0x7f355006cb30 2024-03-29T11:35:10.551 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.517+0000 7f35477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:10.551 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.517+0000 7f35477fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:10.551 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.517+0000 7f35477fe640 10 monclient: dump: 2024-03-29T11:35:10.551 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:35:10.552 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:10.552 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:10.552 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:10.552 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:10.552 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:10.553 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:10.553 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:10.553 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:10.553 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:10.553 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.518+0000 7f35555af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:10.553 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.518+0000 7f35555af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:10.554 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.518+0000 7f35555af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:10.554 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.518+0000 7f35555af640 1 -- 172.21.15.72:0/2626140941 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3534005080 con 0x7f355006cb30 2024-03-29T11:35:10.554 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.518+0000 7f35477fe640 1 -- 172.21.15.72:0/2626140941 <== mon.1 v1:172.21.15.89:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f3540004e10 con 0x7f355006cb30 2024-03-29T11:35:10.554 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.519+0000 7f35477fe640 1 -- 172.21.15.72:0/2626140941 <== mon.1 v1:172.21.15.89:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f3540084ad0 con 0x7f355006cb30 2024-03-29T11:35:10.554 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.519+0000 7f35477fe640 10 client.4243.objecter ms_dispatch 0x7f3550000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:10.555 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.519+0000 7f35477fe640 3 client.4243.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:10.555 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.519+0000 7f35477fe640 3 client.4243.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:10.555 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.519+0000 7f35477fe640 20 client.4243.objecter dump_active .. 0 homeless 2024-03-29T11:35:10.555 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.519+0000 7f3547fff640 10 monclient: get_auth_request con 0x7f3530076580 auth_method 0 2024-03-29T11:35:10.555 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.520+0000 7f35477fe640 10 client.4243.objecter ms_handle_connect 0x7f3530076580 2024-03-29T11:35:10.556 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.525+0000 7f35477fe640 1 -- 172.21.15.72:0/2626140941 <== mon.1 v1:172.21.15.89:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f35400503b0 con 0x7f355006cb30 2024-03-29T11:35:10.556 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.525+0000 7f35477fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:10.556 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.525+0000 7f35477fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:10.875 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.877+0000 7f35555af640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:10.875 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.877+0000 7f35555af640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:10.875 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.877+0000 7f35555af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:10.875 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.877+0000 7f35555af640 1 -- 172.21.15.72:0/2626140941 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3534005330 con 0x7f355006cb30 2024-03-29T11:35:10.875 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.878+0000 7f35477fe640 1 -- 172.21.15.72:0/2626140941 <== mon.1 v1:172.21.15.89:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f35400522e0 con 0x7f355006cb30 2024-03-29T11:35:10.875 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.878+0000 7f35477fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:10.876 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.878+0000 7f35477fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:10.876 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:35:10.876 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:10.879 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.881+0000 7f35555af640 10 librados: watch_flush enter 2024-03-29T11:35:10.879 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.881+0000 7f35555af640 10 librados: watch_flush exit 2024-03-29T11:35:10.879 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.881+0000 7f35555af640 20 client.4243.objecter shutdown clearing up homeless session... 2024-03-29T11:35:10.879 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.881+0000 7f35555af640 10 client.4243.objecter successfully canceled tick 2024-03-29T11:35:10.879 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.881+0000 7f35555af640 1 -- 172.21.15.72:0/2626140941 >> v1:172.21.15.72:6816/1578866102 conn(0x7f3530076580 legacy=0x7f3530078a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:10.879 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.881+0000 7f35555af640 10 monclient: shutdown 2024-03-29T11:35:10.879 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.881+0000 7f35555af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:10.879 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.881+0000 7f35555af640 1 -- 172.21.15.72:0/2626140941 >> v1:172.21.15.89:6789/0 conn(0x7f355006cb30 legacy=0x7f35501d7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:10.879 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.881+0000 7f35555af640 1 -- 172.21.15.72:0/2626140941 shutdown_connections 2024-03-29T11:35:10.880 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.881+0000 7f35555af640 1 -- 172.21.15.72:0/2626140941 >> 172.21.15.72:0/2626140941 conn(0x7f35500b5a20 msgr2=0x7f35500b4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:10.880 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.881+0000 7f35555af640 1 -- 172.21.15.72:0/2626140941 shutdown_connections 2024-03-29T11:35:10.880 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.881+0000 7f35555af640 1 -- 172.21.15.72:0/2626140941 wait complete. 2024-03-29T11:35:10.880 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:10.881+0000 7f35555af640 1 librados: shutdown 2024-03-29T11:35:10.895 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:17.701 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:17.846 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.847+0000 7f4dfcf08640 10 monclient: get_monmap_and_config 2024-03-29T11:35:17.846 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.847+0000 7f4dfcf08640 10 monclient: build_initial_monmap 2024-03-29T11:35:17.846 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.848+0000 7f4dfcf08640 10 monclient: monmap: 2024-03-29T11:35:17.846 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:35:17.846 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:17.846 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:35:17.848944+0000 2024-03-29T11:35:17.846 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:35:17.848944+0000 2024-03-29T11:35:17.846 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:17.846 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:17.847 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:35:17.847 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:35:17.847 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:35:17.847 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:17.848 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.848+0000 7f4dfcf08640 1 Processor -- start 2024-03-29T11:35:17.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.848+0000 7f4dfcf08640 1 -- start start 2024-03-29T11:35:17.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.848+0000 7f4dfcf08640 10 monclient: init 2024-03-29T11:35:17.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.850+0000 7f4dfcf08640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:17.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.850+0000 7f4dfcf08640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:35:17.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.850+0000 7f4dfcf08640 10 monclient(hunting): picked mon.noname-b con 0x7f4df81422b0 addr v1:172.21.15.72:6790/0 2024-03-29T11:35:17.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.850+0000 7f4dfcf08640 10 monclient(hunting): picked mon.noname-a con 0x7f4df8142690 addr v1:172.21.15.72:6789/0 2024-03-29T11:35:17.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.850+0000 7f4dfcf08640 10 monclient(hunting): picked mon.noname-c con 0x7f4df80b17d0 addr v1:172.21.15.89:6789/0 2024-03-29T11:35:17.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.850+0000 7f4dfcf08640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4df8142c80 con 0x7f4df8142690 2024-03-29T11:35:17.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.850+0000 7f4df6d9d640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f4df80b17d0 0x7f4df80b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:34836/0 (socket says 172.21.15.72:34836) 2024-03-29T11:35:17.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.850+0000 7f4df6d9d640 1 -- 172.21.15.72:0/1184683700 learned_addr learned my addr 172.21.15.72:0/1184683700 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:35:17.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.851+0000 7f4dfcf08640 1 -- 172.21.15.72:0/1184683700 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4df80aae40 con 0x7f4df80b17d0 2024-03-29T11:35:17.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.851+0000 7f4dfcf08640 1 -- 172.21.15.72:0/1184683700 --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4df813fbd0 con 0x7f4df81422b0 2024-03-29T11:35:17.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.851+0000 7f4dfcf08640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:17.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.851+0000 7f4dfcf08640 10 monclient(hunting): authenticate will time out at 877.139316s 2024-03-29T11:35:17.854 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.851+0000 7f4df559a640 1 -- 172.21.15.72:0/1184683700 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2665492100 0 0) 0x7f4df813fbd0 con 0x7f4df81422b0 2024-03-29T11:35:17.854 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.851+0000 7f4df559a640 10 monclient(hunting): _negotiate 2024-03-29T11:35:17.854 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.851+0000 7f4df559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:17.854 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.851+0000 7f4df559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:17.854 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.851+0000 7f4df559a640 10 monclient(hunting): my global_id is 4235 2024-03-29T11:35:17.854 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.851+0000 7f4df559a640 1 -- 172.21.15.72:0/1184683700 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4de0003520 con 0x7f4df81422b0 2024-03-29T11:35:17.854 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.851+0000 7f4df559a640 1 -- 172.21.15.72:0/1184683700 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1409509371 0 0) 0x7f4df8142c80 con 0x7f4df8142690 2024-03-29T11:35:17.854 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.851+0000 7f4df559a640 10 monclient(hunting): _negotiate 2024-03-29T11:35:17.854 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.851+0000 7f4df559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:17.855 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.851+0000 7f4df559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:17.855 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.851+0000 7f4df559a640 10 monclient(hunting): my global_id is 4236 2024-03-29T11:35:17.855 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.851+0000 7f4df559a640 1 -- 172.21.15.72:0/1184683700 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4df813fbd0 con 0x7f4df8142690 2024-03-29T11:35:17.855 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.851+0000 7f4df559a640 1 -- 172.21.15.72:0/1184683700 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4222630131 0 0) 0x7f4df80aae40 con 0x7f4df80b17d0 2024-03-29T11:35:17.855 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.851+0000 7f4df559a640 10 monclient(hunting): _negotiate 2024-03-29T11:35:17.855 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.851+0000 7f4df559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:17.855 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.851+0000 7f4df559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:17.855 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.851+0000 7f4df559a640 10 monclient(hunting): my global_id is 4246 2024-03-29T11:35:17.855 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.851+0000 7f4df559a640 1 -- 172.21.15.72:0/1184683700 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4df8142c80 con 0x7f4df80b17d0 2024-03-29T11:35:17.855 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.852+0000 7f4df559a640 1 -- 172.21.15.72:0/1184683700 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4237108835 0 0) 0x7f4de0003520 con 0x7f4df81422b0 2024-03-29T11:35:17.855 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.852+0000 7f4df559a640 1 -- 172.21.15.72:0/1184683700 >> v1:172.21.15.89:6789/0 conn(0x7f4df80b17d0 legacy=0x7f4df80b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:17.855 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.852+0000 7f4df559a640 1 -- 172.21.15.72:0/1184683700 >> v1:172.21.15.72:6789/0 conn(0x7f4df8142690 legacy=0x7f4df80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:17.855 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.852+0000 7f4df559a640 10 monclient: _finish_hunting 0 2024-03-29T11:35:17.855 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.852+0000 7f4df559a640 1 monclient: found mon.noname-b 2024-03-29T11:35:17.856 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.852+0000 7f4df559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:17.856 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.852+0000 7f4df559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.72:6790/0 2024-03-29T11:35:17.856 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.852+0000 7f4df559a640 1 -- 172.21.15.72:0/1184683700 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df814f920 con 0x7f4df81422b0 2024-03-29T11:35:17.856 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.852+0000 7f4df559a640 10 monclient: _finish_auth 0 2024-03-29T11:35:17.856 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.852+0000 7f4df559a640 10 monclient: _check_auth_tickets 2024-03-29T11:35:17.856 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.852+0000 7f4df559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:17.856 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.852+0000 7f4df559a640 1 -- 172.21.15.72:0/1184683700 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f4de8001690 con 0x7f4df81422b0 2024-03-29T11:35:17.856 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.852+0000 7f4df559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:17.856 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.852+0000 7f4df559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:17.856 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.852+0000 7f4df559a640 10 monclient: dump: 2024-03-29T11:35:17.856 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:35:17.856 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:17.856 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:17.856 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:17.856 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:17.856 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:17.857 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:17.857 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:17.857 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:17.857 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:17.857 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.852+0000 7f4dfcf08640 5 monclient: authenticate success, global_id 4235 2024-03-29T11:35:17.857 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.852+0000 7f4dfcf08640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:17.857 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.852+0000 7f4df559a640 1 -- 172.21.15.72:0/1184683700 <== mon.2 v1:172.21.15.72:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4de8001870 con 0x7f4df81422b0 2024-03-29T11:35:17.857 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.852+0000 7f4df559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:17.857 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.852+0000 7f4df559a640 1 -- 172.21.15.72:0/1184683700 <== mon.2 v1:172.21.15.72:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f4de8001c00 con 0x7f4df81422b0 2024-03-29T11:35:17.857 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.852+0000 7f4df559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:17.857 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.852+0000 7f4df559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:17.857 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.852+0000 7f4df559a640 10 monclient: dump: 2024-03-29T11:35:17.857 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:35:17.857 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:17.858 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:17.858 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:17.858 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:17.858 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:17.858 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:17.858 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:17.858 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:17.858 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:17.858 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.852+0000 7f4dfcf08640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:17.858 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.852+0000 7f4dfcf08640 4 set_mon_vals no callback set 2024-03-29T11:35:17.858 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.853+0000 7f4dfcf08640 10 monclient: shutdown 2024-03-29T11:35:17.858 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.853+0000 7f4dfcf08640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:17.858 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.853+0000 7f4dfcf08640 1 -- 172.21.15.72:0/1184683700 >> v1:172.21.15.72:6790/0 conn(0x7f4df81422b0 legacy=0x7f4df813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:17.858 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.853+0000 7f4dfcf08640 1 -- 172.21.15.72:0/1184683700 shutdown_connections 2024-03-29T11:35:17.858 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.853+0000 7f4dfcf08640 1 -- 172.21.15.72:0/1184683700 >> 172.21.15.72:0/1184683700 conn(0x7f4df8066a00 msgr2=0x7f4df8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:17.859 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.853+0000 7f4dfcf08640 1 -- 172.21.15.72:0/1184683700 shutdown_connections 2024-03-29T11:35:17.859 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.853+0000 7f4dfcf08640 1 -- 172.21.15.72:0/1184683700 wait complete. 2024-03-29T11:35:17.859 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.855+0000 7f4dfcf08640 10 monclient: build_initial_monmap 2024-03-29T11:35:17.859 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.855+0000 7f4dfcf08640 10 monclient: monmap: 2024-03-29T11:35:17.859 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:35:17.859 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:17.859 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:35:17.859 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:35:17.859 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:17.859 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:17.859 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:35:17.859 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:35:17.859 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:35:17.859 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:17.859 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.855+0000 7f4dfcf08640 1 librados: starting msgr at 2024-03-29T11:35:17.859 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.855+0000 7f4dfcf08640 1 librados: starting objecter 2024-03-29T11:35:17.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.855+0000 7f4dfcf08640 1 Processor -- start 2024-03-29T11:35:17.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.855+0000 7f4dfcf08640 1 -- start start 2024-03-29T11:35:17.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.855+0000 7f4dfcf08640 1 librados: setting wanted keys 2024-03-29T11:35:17.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.855+0000 7f4dfcf08640 1 librados: calling monclient init 2024-03-29T11:35:17.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.855+0000 7f4dfcf08640 10 monclient: init 2024-03-29T11:35:17.866 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.857+0000 7f4dfcf08640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:17.866 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.857+0000 7f4dfcf08640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:35:17.867 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.857+0000 7f4dfcf08640 10 monclient(hunting): picked mon.noname-c con 0x7f4df80b17d0 addr v1:172.21.15.72:6790/0 2024-03-29T11:35:17.867 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.857+0000 7f4df659c640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f4df80b17d0 0x7f4df81301a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:40944/0 (socket says 172.21.15.72:40944) 2024-03-29T11:35:17.867 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.857+0000 7f4dfcf08640 10 monclient(hunting): picked mon.noname-a con 0x7f4df81422b0 addr v1:172.21.15.72:6789/0 2024-03-29T11:35:17.867 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.857+0000 7f4dfcf08640 10 monclient(hunting): picked mon.noname-b con 0x7f4df8142690 addr v1:172.21.15.89:6789/0 2024-03-29T11:35:17.867 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.857+0000 7f4dfcf08640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4df812f170 con 0x7f4df81422b0 2024-03-29T11:35:17.867 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.857+0000 7f4dfcf08640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4df812f450 con 0x7f4df8142690 2024-03-29T11:35:17.867 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.857+0000 7f4dfcf08640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4df812f730 con 0x7f4df80b17d0 2024-03-29T11:35:17.867 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.857+0000 7f4dfcf08640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:17.867 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.857+0000 7f4dfcf08640 10 monclient(hunting): authenticate will time out at 877.145847s 2024-03-29T11:35:17.867 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.857+0000 7f4df659c640 1 -- 172.21.15.72:0/983387196 learned_addr learned my addr 172.21.15.72:0/983387196 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:35:17.867 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.858+0000 7f4de6ffd640 10 client.?.objecter ms_handle_connect 0x7f4df80b17d0 2024-03-29T11:35:17.867 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.858+0000 7f4de6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:17.867 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.858+0000 7f4de6ffd640 1 -- 172.21.15.72:0/983387196 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3827368806 0 0) 0x7f4df812f730 con 0x7f4df80b17d0 2024-03-29T11:35:17.867 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.858+0000 7f4de6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:17.868 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.858+0000 7f4de6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:17.868 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.858+0000 7f4de6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:17.868 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.858+0000 7f4de6ffd640 10 monclient(hunting): my global_id is 4238 2024-03-29T11:35:17.868 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.858+0000 7f4de6ffd640 1 -- 172.21.15.72:0/983387196 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dd8004eb0 con 0x7f4df80b17d0 2024-03-29T11:35:17.868 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.858+0000 7f4de6ffd640 10 client.?.objecter ms_handle_connect 0x7f4df81422b0 2024-03-29T11:35:17.868 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.858+0000 7f4de6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:17.868 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.858+0000 7f4de6ffd640 1 -- 172.21.15.72:0/983387196 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1135962301 0 0) 0x7f4dd8004eb0 con 0x7f4df80b17d0 2024-03-29T11:35:17.868 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.858+0000 7f4de6ffd640 1 -- 172.21.15.72:0/983387196 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4df812f730 con 0x7f4df80b17d0 2024-03-29T11:35:17.868 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.859+0000 7f4de6ffd640 10 client.?.objecter ms_handle_connect 0x7f4df8142690 2024-03-29T11:35:17.868 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.859+0000 7f4de6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:17.868 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.859+0000 7f4de6ffd640 1 -- 172.21.15.72:0/983387196 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f4de8001c00 con 0x7f4df80b17d0 2024-03-29T11:35:17.868 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.859+0000 7f4de6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:17.868 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.859+0000 7f4de6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:17.868 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.859+0000 7f4de6ffd640 10 monclient(hunting): dump: 2024-03-29T11:35:17.868 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:35:17.869 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:17.869 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:17.869 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:17.869 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:17.869 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:17.869 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:17.869 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:17.869 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:17.869 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:17.869 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.859+0000 7f4de6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:17.869 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.859+0000 7f4de6ffd640 1 -- 172.21.15.72:0/983387196 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 566321551 0 0) 0x7f4df812f170 con 0x7f4df81422b0 2024-03-29T11:35:17.869 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.859+0000 7f4de6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:17.869 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.859+0000 7f4de6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:17.869 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.859+0000 7f4de6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:17.869 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.859+0000 7f4de6ffd640 10 monclient(hunting): my global_id is 4239 2024-03-29T11:35:17.869 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.859+0000 7f4de6ffd640 1 -- 172.21.15.72:0/983387196 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dd8004eb0 con 0x7f4df81422b0 2024-03-29T11:35:17.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.859+0000 7f4de6ffd640 1 -- 172.21.15.72:0/983387196 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3441492512 0 0) 0x7f4df812f450 con 0x7f4df8142690 2024-03-29T11:35:17.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.859+0000 7f4de6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:17.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.859+0000 7f4de6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:17.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.859+0000 7f4de6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:17.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.859+0000 7f4de6ffd640 10 monclient(hunting): my global_id is 4249 2024-03-29T11:35:17.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.859+0000 7f4de6ffd640 1 -- 172.21.15.72:0/983387196 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4df812f170 con 0x7f4df8142690 2024-03-29T11:35:17.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.859+0000 7f4de6ffd640 1 -- 172.21.15.72:0/983387196 <== mon.2 v1:172.21.15.72:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 10523579 0 0) 0x7f4df812f730 con 0x7f4df80b17d0 2024-03-29T11:35:17.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.859+0000 7f4de6ffd640 1 -- 172.21.15.72:0/983387196 >> v1:172.21.15.89:6789/0 conn(0x7f4df8142690 legacy=0x7f4df814da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:17.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.860+0000 7f4de6ffd640 1 -- 172.21.15.72:0/983387196 >> v1:172.21.15.72:6789/0 conn(0x7f4df81422b0 legacy=0x7f4df812dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:17.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.860+0000 7f4de6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:35:17.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.860+0000 7f4de6ffd640 1 monclient: found mon.c 2024-03-29T11:35:17.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.860+0000 7f4de6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:17.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.860+0000 7f4de6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:35:17.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.860+0000 7f4de6ffd640 1 -- 172.21.15.72:0/983387196 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df8130a30 con 0x7f4df80b17d0 2024-03-29T11:35:17.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.860+0000 7f4de6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:35:17.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.860+0000 7f4de6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:35:17.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.860+0000 7f4de6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:17.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.860+0000 7f4dfcf08640 5 monclient: authenticate success, global_id 4238 2024-03-29T11:35:17.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.860+0000 7f4dfcf08640 10 monclient: _renew_subs 2024-03-29T11:35:17.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.860+0000 7f4dfcf08640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:35:17.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.860+0000 7f4dfcf08640 1 -- 172.21.15.72:0/983387196 --> v1:172.21.15.72:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4df812edc0 con 0x7f4df80b17d0 2024-03-29T11:35:17.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.860+0000 7f4dfcf08640 10 client.4238.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:17.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.860+0000 7f4dfcf08640 10 monclient: _renew_subs 2024-03-29T11:35:17.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.860+0000 7f4dfcf08640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:35:17.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.860+0000 7f4dfcf08640 1 -- 172.21.15.72:0/983387196 --> v1:172.21.15.72:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4df812e500 con 0x7f4df80b17d0 2024-03-29T11:35:17.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.860+0000 7f4dfcf08640 1 librados: init done 2024-03-29T11:35:17.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.860+0000 7f4de6ffd640 1 -- 172.21.15.72:0/983387196 <== mon.2 v1:172.21.15.72:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4de80038a0 con 0x7f4df80b17d0 2024-03-29T11:35:17.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.860+0000 7f4de6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:17.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.860+0000 7f4de6ffd640 1 -- 172.21.15.72:0/983387196 <== mon.2 v1:172.21.15.72:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f4de8003c30 con 0x7f4df80b17d0 2024-03-29T11:35:17.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.860+0000 7f4de6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:17.872 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.860+0000 7f4de6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:17.872 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.860+0000 7f4de6ffd640 10 monclient: dump: 2024-03-29T11:35:17.872 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:35:17.872 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:17.872 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:17.872 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:17.872 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:17.872 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:17.872 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:17.872 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:17.872 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:17.872 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:17.872 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.861+0000 7f4df4d99640 4 set_mon_vals no callback set 2024-03-29T11:35:17.872 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.863+0000 7f4de6ffd640 1 -- 172.21.15.72:0/983387196 <== mon.2 v1:172.21.15.72:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f4de8003d90 con 0x7f4df80b17d0 2024-03-29T11:35:17.872 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.863+0000 7f4dfcf08640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:17.873 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.863+0000 7f4dfcf08640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:17.873 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.863+0000 7f4dfcf08640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:35:17.873 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.863+0000 7f4dfcf08640 1 -- 172.21.15.72:0/983387196 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ddc005080 con 0x7f4df80b17d0 2024-03-29T11:35:17.873 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.863+0000 7f4df5d9b640 10 monclient: get_auth_request con 0x7f4dd806dc20 auth_method 0 2024-03-29T11:35:17.873 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.864+0000 7f4de6ffd640 1 -- 172.21.15.72:0/983387196 <== mon.2 v1:172.21.15.72:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f4de8084b80 con 0x7f4df80b17d0 2024-03-29T11:35:17.873 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.864+0000 7f4de6ffd640 10 client.4238.objecter ms_dispatch 0x7f4df8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:17.873 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.864+0000 7f4de6ffd640 3 client.4238.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:17.873 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.864+0000 7f4de6ffd640 3 client.4238.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:17.873 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.864+0000 7f4de6ffd640 20 client.4238.objecter dump_active .. 0 homeless 2024-03-29T11:35:17.873 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.864+0000 7f4de6ffd640 10 client.4238.objecter ms_handle_connect 0x7f4dd806dc20 2024-03-29T11:35:17.874 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.872+0000 7f4de6ffd640 1 -- 172.21.15.72:0/983387196 <== mon.2 v1:172.21.15.72:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4de80504c0 con 0x7f4df80b17d0 2024-03-29T11:35:17.874 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.873+0000 7f4de6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:17.874 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:17.873+0000 7f4de6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:18.182 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:18.184+0000 7f4dfcf08640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:18.182 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:18.184+0000 7f4dfcf08640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:18.182 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:18.184+0000 7f4dfcf08640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:35:18.182 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:18.184+0000 7f4dfcf08640 1 -- 172.21.15.72:0/983387196 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ddc005330 con 0x7f4df80b17d0 2024-03-29T11:35:18.182 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:18.184+0000 7f4de6ffd640 1 -- 172.21.15.72:0/983387196 <== mon.2 v1:172.21.15.72:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f4de80523f0 con 0x7f4df80b17d0 2024-03-29T11:35:18.183 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:18.184+0000 7f4de6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:18.183 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:18.184+0000 7f4de6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:18.183 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:35:18.183 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:18.186 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:18.187+0000 7f4dfcf08640 10 librados: watch_flush enter 2024-03-29T11:35:18.186 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:18.187+0000 7f4dfcf08640 10 librados: watch_flush exit 2024-03-29T11:35:18.186 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:18.187+0000 7f4dfcf08640 20 client.4238.objecter shutdown clearing up homeless session... 2024-03-29T11:35:18.186 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:18.187+0000 7f4dfcf08640 10 client.4238.objecter successfully canceled tick 2024-03-29T11:35:18.186 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:18.188+0000 7f4dfcf08640 1 -- 172.21.15.72:0/983387196 >> v1:172.21.15.72:6816/1578866102 conn(0x7f4dd806dc20 legacy=0x7f4dd80700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:18.186 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:18.188+0000 7f4dfcf08640 10 monclient: shutdown 2024-03-29T11:35:18.186 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:18.188+0000 7f4dfcf08640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:18.186 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:18.188+0000 7f4dfcf08640 1 -- 172.21.15.72:0/983387196 >> v1:172.21.15.72:6790/0 conn(0x7f4df80b17d0 legacy=0x7f4df81301a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:18.186 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:18.188+0000 7f4dfcf08640 1 -- 172.21.15.72:0/983387196 shutdown_connections 2024-03-29T11:35:18.187 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:18.188+0000 7f4dfcf08640 1 -- 172.21.15.72:0/983387196 >> 172.21.15.72:0/983387196 conn(0x7f4df8066a00 msgr2=0x7f4df813f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:18.187 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:18.188+0000 7f4dfcf08640 1 -- 172.21.15.72:0/983387196 shutdown_connections 2024-03-29T11:35:18.187 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:18.188+0000 7f4dfcf08640 1 -- 172.21.15.72:0/983387196 wait complete. 2024-03-29T11:35:18.187 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:18.188+0000 7f4dfcf08640 1 librados: shutdown 2024-03-29T11:35:18.200 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:25.007 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:25.136 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.138+0000 7f59ab227640 10 monclient: get_monmap_and_config 2024-03-29T11:35:25.136 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.138+0000 7f59ab227640 10 monclient: build_initial_monmap 2024-03-29T11:35:25.136 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.138+0000 7f59ab227640 10 monclient: monmap: 2024-03-29T11:35:25.136 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:35:25.136 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:25.136 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:35:25.138992+0000 2024-03-29T11:35:25.136 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:35:25.138992+0000 2024-03-29T11:35:25.136 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:25.136 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:25.136 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:35:25.136 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:35:25.136 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:35:25.137 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:25.137 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.138+0000 7f59ab227640 1 Processor -- start 2024-03-29T11:35:25.137 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.139+0000 7f59ab227640 1 -- start start 2024-03-29T11:35:25.137 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.139+0000 7f59ab227640 10 monclient: init 2024-03-29T11:35:25.137 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.139+0000 7f59ab227640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:25.138 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.139+0000 7f59ab227640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:35:25.138 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.139+0000 7f59ab227640 10 monclient(hunting): picked mon.noname-c con 0x7f59a41422b0 addr v1:172.21.15.89:6789/0 2024-03-29T11:35:25.138 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.139+0000 7f59ab227640 10 monclient(hunting): picked mon.noname-a con 0x7f59a4142690 addr v1:172.21.15.72:6789/0 2024-03-29T11:35:25.138 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.139+0000 7f59ab227640 10 monclient(hunting): picked mon.noname-b con 0x7f59a40b17d0 addr v1:172.21.15.72:6790/0 2024-03-29T11:35:25.138 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.139+0000 7f59ab227640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a4142c80 con 0x7f59a4142690 2024-03-29T11:35:25.138 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.139+0000 7f59ab227640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a40aae40 con 0x7f59a41422b0 2024-03-29T11:35:25.138 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.139+0000 7f59ab227640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a413fbd0 con 0x7f59a40b17d0 2024-03-29T11:35:25.138 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.139+0000 7f59ab227640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:25.138 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.139+0000 7f59ab227640 10 monclient(hunting): authenticate will time out at 884.427940s 2024-03-29T11:35:25.138 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.140+0000 7f59a97c4640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f59a40b17d0 0x7f59a40b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:52122/0 (socket says 172.21.15.72:52122) 2024-03-29T11:35:25.138 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.140+0000 7f59a97c4640 1 -- 172.21.15.72:0/1751846738 learned_addr learned my addr 172.21.15.72:0/1751846738 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:35:25.144 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.141+0000 7f599b7fe640 1 -- 172.21.15.72:0/1751846738 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2180868486 0 0) 0x7f59a413fbd0 con 0x7f59a40b17d0 2024-03-29T11:35:25.144 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.141+0000 7f599b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:25.144 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.141+0000 7f599b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:25.144 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.141+0000 7f599b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:25.144 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.141+0000 7f599b7fe640 10 monclient(hunting): my global_id is 4241 2024-03-29T11:35:25.144 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.141+0000 7f599b7fe640 1 -- 172.21.15.72:0/1751846738 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f598c003520 con 0x7f59a40b17d0 2024-03-29T11:35:25.144 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.141+0000 7f599b7fe640 1 -- 172.21.15.72:0/1751846738 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 70159815 0 0) 0x7f598c003520 con 0x7f59a40b17d0 2024-03-29T11:35:25.145 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.141+0000 7f599b7fe640 1 -- 172.21.15.72:0/1751846738 >> v1:172.21.15.89:6789/0 conn(0x7f59a41422b0 legacy=0x7f59a413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:35:25.145 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.141+0000 7f599b7fe640 1 -- 172.21.15.72:0/1751846738 >> v1:172.21.15.72:6789/0 conn(0x7f59a4142690 legacy=0x7f59a40b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:35:25.145 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.141+0000 7f599b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:35:25.145 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.141+0000 7f599b7fe640 1 monclient: found mon.noname-b 2024-03-29T11:35:25.145 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.141+0000 7f599b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:25.145 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.141+0000 7f599b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.72:6790/0 2024-03-29T11:35:25.145 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.141+0000 7f599b7fe640 1 -- 172.21.15.72:0/1751846738 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a4151940 con 0x7f59a40b17d0 2024-03-29T11:35:25.145 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.142+0000 7f599b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:35:25.145 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.142+0000 7f599b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:35:25.145 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.142+0000 7f599b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:25.145 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.142+0000 7f599b7fe640 1 -- 172.21.15.72:0/1751846738 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f599c001a10 con 0x7f59a40b17d0 2024-03-29T11:35:25.145 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.142+0000 7f599b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:25.145 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.142+0000 7f599b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:25.145 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.142+0000 7f599b7fe640 10 monclient: dump: 2024-03-29T11:35:25.145 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:35:25.146 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:25.146 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:25.146 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:25.146 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:25.146 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:25.146 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:25.146 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:25.146 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:25.146 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:25.146 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.142+0000 7f59ab227640 5 monclient: authenticate success, global_id 4241 2024-03-29T11:35:25.146 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.142+0000 7f59ab227640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:25.146 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.142+0000 7f599b7fe640 1 -- 172.21.15.72:0/1751846738 <== mon.2 v1:172.21.15.72:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f599c001b70 con 0x7f59a40b17d0 2024-03-29T11:35:25.146 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.142+0000 7f599b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:25.146 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.142+0000 7f59ab227640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:25.146 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.142+0000 7f59ab227640 4 set_mon_vals no callback set 2024-03-29T11:35:25.147 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.143+0000 7f59ab227640 10 monclient: shutdown 2024-03-29T11:35:25.147 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.143+0000 7f59ab227640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:25.147 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.143+0000 7f59ab227640 1 -- 172.21.15.72:0/1751846738 >> v1:172.21.15.72:6790/0 conn(0x7f59a40b17d0 legacy=0x7f59a40b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:25.147 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.143+0000 7f59ab227640 1 -- 172.21.15.72:0/1751846738 shutdown_connections 2024-03-29T11:35:25.147 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.143+0000 7f59ab227640 1 -- 172.21.15.72:0/1751846738 >> 172.21.15.72:0/1751846738 conn(0x7f59a4066a00 msgr2=0x7f59a4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:25.147 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.144+0000 7f59ab227640 1 -- 172.21.15.72:0/1751846738 shutdown_connections 2024-03-29T11:35:25.147 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.144+0000 7f59ab227640 1 -- 172.21.15.72:0/1751846738 wait complete. 2024-03-29T11:35:25.147 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.144+0000 7f59ab227640 10 monclient: build_initial_monmap 2024-03-29T11:35:25.147 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.144+0000 7f59ab227640 10 monclient: monmap: 2024-03-29T11:35:25.147 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:35:25.147 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:25.147 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:35:25.147 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:35:25.147 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:25.147 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:25.148 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:35:25.148 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:35:25.148 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:35:25.148 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:25.148 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.144+0000 7f59ab227640 1 librados: starting msgr at 2024-03-29T11:35:25.148 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.144+0000 7f59ab227640 1 librados: starting objecter 2024-03-29T11:35:25.148 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.145+0000 7f59ab227640 1 Processor -- start 2024-03-29T11:35:25.148 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.145+0000 7f59ab227640 1 -- start start 2024-03-29T11:35:25.148 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.145+0000 7f59ab227640 1 librados: setting wanted keys 2024-03-29T11:35:25.148 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.145+0000 7f59ab227640 1 librados: calling monclient init 2024-03-29T11:35:25.148 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.145+0000 7f59ab227640 10 monclient: init 2024-03-29T11:35:25.148 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.146+0000 7f59ab227640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:25.148 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.146+0000 7f59ab227640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:35:25.148 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.146+0000 7f59ab227640 10 monclient(hunting): picked mon.noname-b con 0x7f59a40b17d0 addr v1:172.21.15.89:6789/0 2024-03-29T11:35:25.148 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.146+0000 7f59ab227640 10 monclient(hunting): picked mon.noname-c con 0x7f59a41422b0 addr v1:172.21.15.72:6790/0 2024-03-29T11:35:25.148 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.146+0000 7f59ab227640 10 monclient(hunting): picked mon.noname-a con 0x7f59a4142690 addr v1:172.21.15.72:6789/0 2024-03-29T11:35:25.149 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.146+0000 7f59ab227640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a41dad60 con 0x7f59a4142690 2024-03-29T11:35:25.149 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.146+0000 7f59ab227640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a40ad130 con 0x7f59a40b17d0 2024-03-29T11:35:25.149 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.146+0000 7f59ab227640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a40b71b0 con 0x7f59a41422b0 2024-03-29T11:35:25.149 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.146+0000 7f59ab227640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:25.149 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.146+0000 7f59ab227640 10 monclient(hunting): authenticate will time out at 884.434284s 2024-03-29T11:35:25.149 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.146+0000 7f599bfff640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f59a41422b0 0x7f59a40ab520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:52718/0 (socket says 172.21.15.72:52718) 2024-03-29T11:35:25.149 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.146+0000 7f599bfff640 1 -- 172.21.15.72:0/1365782409 learned_addr learned my addr 172.21.15.72:0/1365782409 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:35:25.149 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.146+0000 7f59997fa640 10 client.?.objecter ms_handle_connect 0x7f59a41422b0 2024-03-29T11:35:25.149 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.146+0000 7f59997fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:25.149 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.146+0000 7f59997fa640 1 -- 172.21.15.72:0/1365782409 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3905989936 0 0) 0x7f59a40b71b0 con 0x7f59a41422b0 2024-03-29T11:35:25.149 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.146+0000 7f59997fa640 10 monclient(hunting): _negotiate 2024-03-29T11:35:25.149 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.146+0000 7f59997fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:25.149 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.146+0000 7f59997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:25.149 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.146+0000 7f59997fa640 10 monclient(hunting): my global_id is 4244 2024-03-29T11:35:25.149 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.146+0000 7f59997fa640 1 -- 172.21.15.72:0/1365782409 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5990004eb0 con 0x7f59a41422b0 2024-03-29T11:35:25.152 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.147+0000 7f59997fa640 1 -- 172.21.15.72:0/1365782409 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3312997465 0 0) 0x7f5990004eb0 con 0x7f59a41422b0 2024-03-29T11:35:25.152 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.147+0000 7f59997fa640 1 -- 172.21.15.72:0/1365782409 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59a40b71b0 con 0x7f59a41422b0 2024-03-29T11:35:25.152 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.147+0000 7f59997fa640 1 -- 172.21.15.72:0/1365782409 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f59940017d0 con 0x7f59a41422b0 2024-03-29T11:35:25.152 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.147+0000 7f59997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:25.152 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.147+0000 7f59997fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:25.152 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.147+0000 7f59997fa640 10 monclient(hunting): dump: 2024-03-29T11:35:25.152 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:35:25.152 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:25.152 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:25.152 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:25.153 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:25.153 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:25.153 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:25.153 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:25.153 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:25.153 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:25.153 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.147+0000 7f59997fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:25.153 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.147+0000 7f59997fa640 1 -- 172.21.15.72:0/1365782409 <== mon.2 v1:172.21.15.72:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2032191206 0 0) 0x7f59a40b71b0 con 0x7f59a41422b0 2024-03-29T11:35:25.153 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.147+0000 7f59997fa640 1 -- 172.21.15.72:0/1365782409 >> v1:172.21.15.89:6789/0 conn(0x7f59a40b17d0 legacy=0x7f59a40af8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:25.153 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.147+0000 7f59997fa640 1 -- 172.21.15.72:0/1365782409 >> v1:172.21.15.72:6789/0 conn(0x7f59a4142690 legacy=0x7f59a41e30a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:35:25.153 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.149+0000 7f59997fa640 10 monclient: _finish_hunting 0 2024-03-29T11:35:25.153 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.149+0000 7f59997fa640 1 monclient: found mon.c 2024-03-29T11:35:25.153 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.149+0000 7f59997fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:25.153 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.149+0000 7f59997fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:35:25.153 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.149+0000 7f59997fa640 1 -- 172.21.15.72:0/1365782409 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a40b7340 con 0x7f59a41422b0 2024-03-29T11:35:25.154 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.149+0000 7f59997fa640 10 monclient: _finish_auth 0 2024-03-29T11:35:25.154 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.149+0000 7f59997fa640 10 monclient: _check_auth_tickets 2024-03-29T11:35:25.154 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.149+0000 7f59997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:25.154 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.149+0000 7f59ab227640 5 monclient: authenticate success, global_id 4244 2024-03-29T11:35:25.154 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.149+0000 7f59ab227640 10 monclient: _renew_subs 2024-03-29T11:35:25.154 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.149+0000 7f59ab227640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:35:25.154 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.149+0000 7f59ab227640 1 -- 172.21.15.72:0/1365782409 --> v1:172.21.15.72:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59a40b0640 con 0x7f59a41422b0 2024-03-29T11:35:25.154 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.149+0000 7f59ab227640 10 client.4244.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:25.154 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.150+0000 7f59ab227640 10 monclient: _renew_subs 2024-03-29T11:35:25.154 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.150+0000 7f59ab227640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:35:25.154 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.150+0000 7f59ab227640 1 -- 172.21.15.72:0/1365782409 --> v1:172.21.15.72:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f59a4150a40 con 0x7f59a41422b0 2024-03-29T11:35:25.154 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.150+0000 7f59ab227640 1 librados: init done 2024-03-29T11:35:25.154 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.150+0000 7f59997fa640 1 -- 172.21.15.72:0/1365782409 <== mon.2 v1:172.21.15.72:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5994001930 con 0x7f59a41422b0 2024-03-29T11:35:25.155 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.150+0000 7f59997fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:25.155 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.150+0000 7f599affd640 4 set_mon_vals no callback set 2024-03-29T11:35:25.155 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.151+0000 7f59997fa640 1 -- 172.21.15.72:0/1365782409 <== mon.2 v1:172.21.15.72:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f5994004a40 con 0x7f59a41422b0 2024-03-29T11:35:25.155 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.151+0000 7f59997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:25.155 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.151+0000 7f59997fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:25.155 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.151+0000 7f59997fa640 10 monclient: dump: 2024-03-29T11:35:25.155 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:35:25.155 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:25.155 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:25.155 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:25.155 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:25.155 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:25.155 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:25.155 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:25.155 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:25.155 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:25.156 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.151+0000 7f59ab227640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:25.156 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.151+0000 7f59ab227640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:25.156 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.151+0000 7f59ab227640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:35:25.156 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.151+0000 7f59ab227640 1 -- 172.21.15.72:0/1365782409 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5984005080 con 0x7f59a41422b0 2024-03-29T11:35:25.156 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.152+0000 7f59997fa640 1 -- 172.21.15.72:0/1365782409 <== mon.2 v1:172.21.15.72:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f599401aa10 con 0x7f59a41422b0 2024-03-29T11:35:25.156 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.153+0000 7f59a8fc3640 10 monclient: get_auth_request con 0x7f599006dc70 auth_method 0 2024-03-29T11:35:25.156 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.153+0000 7f59997fa640 1 -- 172.21.15.72:0/1365782409 <== mon.2 v1:172.21.15.72:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f5994084a80 con 0x7f59a41422b0 2024-03-29T11:35:25.156 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.153+0000 7f59997fa640 10 client.4244.objecter ms_dispatch 0x7f59a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:25.156 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.153+0000 7f59997fa640 3 client.4244.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:25.156 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.153+0000 7f59997fa640 3 client.4244.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:25.156 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.154+0000 7f59997fa640 20 client.4244.objecter dump_active .. 0 homeless 2024-03-29T11:35:25.156 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.154+0000 7f59997fa640 10 client.4244.objecter ms_handle_connect 0x7f599006dc70 2024-03-29T11:35:25.158 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.160+0000 7f59997fa640 1 -- 172.21.15.72:0/1365782409 <== mon.2 v1:172.21.15.72:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5994050190 con 0x7f59a41422b0 2024-03-29T11:35:25.158 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.160+0000 7f59997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:25.158 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.160+0000 7f59997fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:25.453 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.455+0000 7f59ab227640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:25.454 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:35:25.454 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:25.596 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.455+0000 7f59ab227640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:25.596 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.455+0000 7f59ab227640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:35:25.596 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.455+0000 7f59ab227640 1 -- 172.21.15.72:0/1365782409 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5984005330 con 0x7f59a41422b0 2024-03-29T11:35:25.596 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.456+0000 7f59997fa640 1 -- 172.21.15.72:0/1365782409 <== mon.2 v1:172.21.15.72:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f5994052290 con 0x7f59a41422b0 2024-03-29T11:35:25.596 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.456+0000 7f59997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:25.597 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.456+0000 7f59997fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:25.597 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.459+0000 7f59ab227640 10 librados: watch_flush enter 2024-03-29T11:35:25.597 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.459+0000 7f59ab227640 10 librados: watch_flush exit 2024-03-29T11:35:25.597 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.459+0000 7f59ab227640 20 client.4244.objecter shutdown clearing up homeless session... 2024-03-29T11:35:25.597 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.459+0000 7f59ab227640 10 client.4244.objecter successfully canceled tick 2024-03-29T11:35:25.597 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.460+0000 7f59ab227640 1 -- 172.21.15.72:0/1365782409 >> v1:172.21.15.72:6816/1578866102 conn(0x7f599006dc70 legacy=0x7f5990070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:25.597 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.460+0000 7f59ab227640 10 monclient: shutdown 2024-03-29T11:35:25.597 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.460+0000 7f59ab227640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:25.597 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.460+0000 7f59ab227640 1 -- 172.21.15.72:0/1365782409 >> v1:172.21.15.72:6790/0 conn(0x7f59a41422b0 legacy=0x7f59a40ab520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:25.597 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.460+0000 7f59ab227640 1 -- 172.21.15.72:0/1365782409 shutdown_connections 2024-03-29T11:35:25.597 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.460+0000 7f59ab227640 1 -- 172.21.15.72:0/1365782409 >> 172.21.15.72:0/1365782409 conn(0x7f59a4066a00 msgr2=0x7f59a40b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:25.597 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.460+0000 7f59ab227640 1 -- 172.21.15.72:0/1365782409 shutdown_connections 2024-03-29T11:35:25.597 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.460+0000 7f59ab227640 1 -- 172.21.15.72:0/1365782409 wait complete. 2024-03-29T11:35:25.597 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:25.460+0000 7f59ab227640 1 librados: shutdown 2024-03-29T11:35:25.611 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:32.418 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:32.564 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.566+0000 7f443f251640 10 monclient: get_monmap_and_config 2024-03-29T11:35:32.564 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.566+0000 7f443f251640 10 monclient: build_initial_monmap 2024-03-29T11:35:32.564 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.566+0000 7f443f251640 10 monclient: monmap: 2024-03-29T11:35:32.564 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:35:32.564 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:32.564 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:35:32.567145+0000 2024-03-29T11:35:32.565 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:35:32.567145+0000 2024-03-29T11:35:32.565 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:32.565 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:32.565 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:35:32.565 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:35:32.565 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:35:32.565 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:32.567 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.569+0000 7f443f251640 1 Processor -- start 2024-03-29T11:35:32.568 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.569+0000 7f443f251640 1 -- start start 2024-03-29T11:35:32.568 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.569+0000 7f443f251640 10 monclient: init 2024-03-29T11:35:32.568 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.570+0000 7f443f251640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:32.568 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.570+0000 7f443f251640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:35:32.568 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.570+0000 7f443f251640 10 monclient(hunting): picked mon.noname-a con 0x7f44300389e0 addr v1:172.21.15.72:6789/0 2024-03-29T11:35:32.568 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.570+0000 7f443f251640 10 monclient(hunting): picked mon.noname-c con 0x7f4430038dc0 addr v1:172.21.15.89:6789/0 2024-03-29T11:35:32.568 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.570+0000 7f443f251640 10 monclient(hunting): picked mon.noname-b con 0x7f4430030660 addr v1:172.21.15.72:6790/0 2024-03-29T11:35:32.568 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.570+0000 7f443f251640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44300c54c0 con 0x7f44300389e0 2024-03-29T11:35:32.568 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.570+0000 7f443f251640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4430036430 con 0x7f4430038dc0 2024-03-29T11:35:32.569 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.570+0000 7f443f251640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4430039840 con 0x7f4430030660 2024-03-29T11:35:32.569 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.570+0000 7f443f251640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:32.569 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.570+0000 7f443f251640 10 monclient(hunting): authenticate will time out at 891.858502s 2024-03-29T11:35:32.571 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.573+0000 7f443e24f640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f44300389e0 0x7f4430033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37549/0 (socket says 172.21.15.72:37549) 2024-03-29T11:35:32.571 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.573+0000 7f443e24f640 1 -- 172.21.15.72:0/3471203506 learned_addr learned my addr 172.21.15.72:0/3471203506 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:35:32.577 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.577+0000 7f443d24d640 1 -- 172.21.15.72:0/3471203506 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1702142977 0 0) 0x7f4430036430 con 0x7f4430038dc0 2024-03-29T11:35:32.577 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.577+0000 7f443d24d640 10 monclient(hunting): _negotiate 2024-03-29T11:35:32.578 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.577+0000 7f443d24d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:32.578 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.577+0000 7f443d24d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:32.578 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.577+0000 7f443d24d640 10 monclient(hunting): my global_id is 4252 2024-03-29T11:35:32.578 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.577+0000 7f443d24d640 1 -- 172.21.15.72:0/3471203506 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4424003520 con 0x7f4430038dc0 2024-03-29T11:35:32.578 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.577+0000 7f443d24d640 1 -- 172.21.15.72:0/3471203506 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3625250082 0 0) 0x7f44300c54c0 con 0x7f44300389e0 2024-03-29T11:35:32.578 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.577+0000 7f443d24d640 10 monclient(hunting): _negotiate 2024-03-29T11:35:32.578 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.577+0000 7f443d24d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:32.578 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.577+0000 7f443d24d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:32.578 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.577+0000 7f443d24d640 10 monclient(hunting): my global_id is 4242 2024-03-29T11:35:32.578 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.577+0000 7f443d24d640 1 -- 172.21.15.72:0/3471203506 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4430036430 con 0x7f44300389e0 2024-03-29T11:35:32.578 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.577+0000 7f443d24d640 1 -- 172.21.15.72:0/3471203506 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 967432491 0 0) 0x7f4430039840 con 0x7f4430030660 2024-03-29T11:35:32.578 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.577+0000 7f443d24d640 10 monclient(hunting): _negotiate 2024-03-29T11:35:32.578 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.577+0000 7f443d24d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:32.578 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.577+0000 7f443d24d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:32.578 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.577+0000 7f443d24d640 10 monclient(hunting): my global_id is 4247 2024-03-29T11:35:32.579 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.577+0000 7f443d24d640 1 -- 172.21.15.72:0/3471203506 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44300c54c0 con 0x7f4430030660 2024-03-29T11:35:32.579 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.577+0000 7f443d24d640 1 -- 172.21.15.72:0/3471203506 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3222231316 0 0) 0x7f4424003520 con 0x7f4430038dc0 2024-03-29T11:35:32.579 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.577+0000 7f443d24d640 1 -- 172.21.15.72:0/3471203506 >> v1:172.21.15.72:6790/0 conn(0x7f4430030660 legacy=0x7f44300d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:32.579 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.577+0000 7f443d24d640 1 -- 172.21.15.72:0/3471203506 >> v1:172.21.15.72:6789/0 conn(0x7f44300389e0 legacy=0x7f4430033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:32.579 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.577+0000 7f443d24d640 10 monclient: _finish_hunting 0 2024-03-29T11:35:32.579 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.577+0000 7f443d24d640 1 monclient: found mon.noname-c 2024-03-29T11:35:32.579 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.577+0000 7f443d24d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:32.579 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.577+0000 7f443d24d640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.89:6789/0 2024-03-29T11:35:32.579 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.578+0000 7f443d24d640 1 -- 172.21.15.72:0/3471203506 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4430035690 con 0x7f4430038dc0 2024-03-29T11:35:32.579 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.578+0000 7f443d24d640 10 monclient: _finish_auth 0 2024-03-29T11:35:32.579 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.578+0000 7f443d24d640 10 monclient: _check_auth_tickets 2024-03-29T11:35:32.579 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.578+0000 7f443d24d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:32.579 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.578+0000 7f443d24d640 1 -- 172.21.15.72:0/3471203506 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f4428001690 con 0x7f4430038dc0 2024-03-29T11:35:32.579 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.578+0000 7f443d24d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:32.579 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.578+0000 7f443d24d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:32.579 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.578+0000 7f443d24d640 10 monclient: dump: 2024-03-29T11:35:32.580 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:35:32.580 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:32.580 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:32.580 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:32.580 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:32.580 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:32.580 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:32.580 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:32.580 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:32.580 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:32.580 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.578+0000 7f443d24d640 1 -- 172.21.15.72:0/3471203506 <== mon.1 v1:172.21.15.89:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44280017f0 con 0x7f4430038dc0 2024-03-29T11:35:32.580 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.578+0000 7f443d24d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:32.580 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.578+0000 7f443d24d640 1 -- 172.21.15.72:0/3471203506 <== mon.1 v1:172.21.15.89:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f4428001c00 con 0x7f4430038dc0 2024-03-29T11:35:32.580 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.578+0000 7f443d24d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:32.580 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.578+0000 7f443d24d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:32.581 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.578+0000 7f443d24d640 10 monclient: dump: 2024-03-29T11:35:32.581 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:35:32.581 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:32.581 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:32.581 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:32.581 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:32.581 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:32.581 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:32.581 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:32.581 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:32.581 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:32.581 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.578+0000 7f443f251640 5 monclient: authenticate success, global_id 4252 2024-03-29T11:35:32.581 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.578+0000 7f443f251640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:32.581 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.578+0000 7f443f251640 4 set_mon_vals no callback set 2024-03-29T11:35:32.581 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.579+0000 7f443f251640 10 monclient: shutdown 2024-03-29T11:35:32.581 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.579+0000 7f443f251640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:32.581 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.579+0000 7f443f251640 1 -- 172.21.15.72:0/3471203506 >> v1:172.21.15.89:6789/0 conn(0x7f4430038dc0 legacy=0x7f44300cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:32.582 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.579+0000 7f443f251640 1 -- 172.21.15.72:0/3471203506 shutdown_connections 2024-03-29T11:35:32.582 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.579+0000 7f443f251640 1 -- 172.21.15.72:0/3471203506 >> 172.21.15.72:0/3471203506 conn(0x7f4430005c00 msgr2=0x7f4430006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:32.583 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.580+0000 7f443f251640 1 -- 172.21.15.72:0/3471203506 shutdown_connections 2024-03-29T11:35:32.583 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.580+0000 7f443f251640 1 -- 172.21.15.72:0/3471203506 wait complete. 2024-03-29T11:35:32.583 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.581+0000 7f443f251640 10 monclient: build_initial_monmap 2024-03-29T11:35:32.583 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.581+0000 7f443f251640 10 monclient: monmap: 2024-03-29T11:35:32.583 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:35:32.583 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:32.583 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:35:32.583 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:35:32.584 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:32.584 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:32.584 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:35:32.584 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:35:32.584 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:35:32.584 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:32.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.582+0000 7f443f251640 1 librados: starting msgr at 2024-03-29T11:35:32.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.582+0000 7f443f251640 1 librados: starting objecter 2024-03-29T11:35:32.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.582+0000 7f443f251640 1 Processor -- start 2024-03-29T11:35:32.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.582+0000 7f443f251640 1 -- start start 2024-03-29T11:35:32.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.582+0000 7f443f251640 1 librados: setting wanted keys 2024-03-29T11:35:32.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.582+0000 7f443f251640 1 librados: calling monclient init 2024-03-29T11:35:32.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.582+0000 7f443f251640 10 monclient: init 2024-03-29T11:35:32.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.583+0000 7f443f251640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:32.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.583+0000 7f443f251640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:35:32.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.583+0000 7f443f251640 10 monclient(hunting): picked mon.noname-c con 0x7f4430030660 addr v1:172.21.15.72:6790/0 2024-03-29T11:35:32.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.583+0000 7f443f251640 10 monclient(hunting): picked mon.noname-a con 0x7f443002d2f0 addr v1:172.21.15.72:6789/0 2024-03-29T11:35:32.585 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.583+0000 7f443f251640 10 monclient(hunting): picked mon.noname-b con 0x7f4430152c90 addr v1:172.21.15.89:6789/0 2024-03-29T11:35:32.585 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.583+0000 7f443f251640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44300c8040 con 0x7f443002d2f0 2024-03-29T11:35:32.585 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.583+0000 7f443f251640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4430153270 con 0x7f4430152c90 2024-03-29T11:35:32.585 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.583+0000 7f443f251640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44300c82b0 con 0x7f4430030660 2024-03-29T11:35:32.585 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.583+0000 7f443f251640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:32.585 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.583+0000 7f443f251640 10 monclient(hunting): authenticate will time out at 891.871672s 2024-03-29T11:35:32.587 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.589+0000 7f443e24f640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f4430030660 0x7f44300c5f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:33509/0 (socket says 172.21.15.72:33509) 2024-03-29T11:35:32.587 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.589+0000 7f443e24f640 1 -- 172.21.15.72:0/2719619891 learned_addr learned my addr 172.21.15.72:0/2719619891 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:35:32.589 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.591+0000 7f442effd640 10 client.?.objecter ms_handle_connect 0x7f4430030660 2024-03-29T11:35:32.589 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.591+0000 7f442effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:32.589 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.591+0000 7f442effd640 10 client.?.objecter ms_handle_connect 0x7f4430152c90 2024-03-29T11:35:32.589 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.591+0000 7f442effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:32.589 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.591+0000 7f442effd640 1 -- 172.21.15.72:0/2719619891 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3307228627 0 0) 0x7f44300c82b0 con 0x7f4430030660 2024-03-29T11:35:32.589 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.591+0000 7f442effd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:32.589 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.591+0000 7f442effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:32.589 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.591+0000 7f442effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:32.589 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.591+0000 7f442effd640 10 monclient(hunting): my global_id is 4250 2024-03-29T11:35:32.589 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.591+0000 7f442effd640 1 -- 172.21.15.72:0/2719619891 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f441c004eb0 con 0x7f4430030660 2024-03-29T11:35:32.593 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.592+0000 7f442effd640 1 -- 172.21.15.72:0/2719619891 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1228405714 0 0) 0x7f4430153270 con 0x7f4430152c90 2024-03-29T11:35:32.593 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.592+0000 7f442effd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:32.593 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.592+0000 7f442effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:32.593 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.592+0000 7f442effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:32.593 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.592+0000 7f442effd640 10 monclient(hunting): my global_id is 4255 2024-03-29T11:35:32.593 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.592+0000 7f442effd640 1 -- 172.21.15.72:0/2719619891 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44300c82b0 con 0x7f4430152c90 2024-03-29T11:35:32.593 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.592+0000 7f442effd640 10 client.?.objecter ms_handle_connect 0x7f443002d2f0 2024-03-29T11:35:32.593 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.593+0000 7f442effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:32.594 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.593+0000 7f442effd640 1 -- 172.21.15.72:0/2719619891 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 666928636 0 0) 0x7f44300c82b0 con 0x7f4430152c90 2024-03-29T11:35:32.594 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.593+0000 7f442effd640 1 -- 172.21.15.72:0/2719619891 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4430153270 con 0x7f4430152c90 2024-03-29T11:35:32.594 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.593+0000 7f442effd640 1 -- 172.21.15.72:0/2719619891 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3545900764 0 0) 0x7f44300c8040 con 0x7f443002d2f0 2024-03-29T11:35:32.594 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.593+0000 7f442effd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:32.594 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.593+0000 7f442effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:32.594 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.593+0000 7f442effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:32.594 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.593+0000 7f442effd640 10 monclient(hunting): my global_id is 4245 2024-03-29T11:35:32.594 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.593+0000 7f442effd640 1 -- 172.21.15.72:0/2719619891 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44300c82b0 con 0x7f443002d2f0 2024-03-29T11:35:32.594 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.593+0000 7f442effd640 1 -- 172.21.15.72:0/2719619891 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f44340018b0 con 0x7f4430152c90 2024-03-29T11:35:32.594 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.593+0000 7f442effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:32.594 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.593+0000 7f442effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:32.594 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.593+0000 7f442effd640 10 monclient(hunting): dump: 2024-03-29T11:35:32.594 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:35:32.594 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:32.594 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:32.594 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:32.595 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:32.595 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:32.595 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:32.595 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:32.595 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:32.595 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:32.595 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.593+0000 7f442effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:32.595 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.594+0000 7f442effd640 1 -- 172.21.15.72:0/2719619891 <== mon.1 v1:172.21.15.89:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 127678394 0 0) 0x7f4430153270 con 0x7f4430152c90 2024-03-29T11:35:32.595 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.594+0000 7f442effd640 1 -- 172.21.15.72:0/2719619891 >> v1:172.21.15.72:6790/0 conn(0x7f4430030660 legacy=0x7f44300c5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:32.595 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.594+0000 7f442effd640 1 -- 172.21.15.72:0/2719619891 >> v1:172.21.15.72:6789/0 conn(0x7f443002d2f0 legacy=0x7f44300c6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:32.595 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.594+0000 7f442effd640 10 monclient: _finish_hunting 0 2024-03-29T11:35:32.595 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.594+0000 7f442effd640 1 monclient: found mon.b 2024-03-29T11:35:32.595 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.594+0000 7f442effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:32.595 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.594+0000 7f442effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:32.595 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.594+0000 7f442effd640 1 -- 172.21.15.72:0/2719619891 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44300c8590 con 0x7f4430152c90 2024-03-29T11:35:32.595 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.594+0000 7f442effd640 10 monclient: _finish_auth 0 2024-03-29T11:35:32.595 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.594+0000 7f442effd640 10 monclient: _check_auth_tickets 2024-03-29T11:35:32.596 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.594+0000 7f442effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:32.596 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.594+0000 7f443f251640 5 monclient: authenticate success, global_id 4255 2024-03-29T11:35:32.596 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.594+0000 7f443f251640 10 monclient: _renew_subs 2024-03-29T11:35:32.596 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.594+0000 7f443f251640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:32.596 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.594+0000 7f443f251640 1 -- 172.21.15.72:0/2719619891 --> v1:172.21.15.89:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44300c71e0 con 0x7f4430152c90 2024-03-29T11:35:32.596 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.595+0000 7f442effd640 1 -- 172.21.15.72:0/2719619891 <== mon.1 v1:172.21.15.89:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4434003610 con 0x7f4430152c90 2024-03-29T11:35:32.596 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.595+0000 7f442effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:32.596 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.595+0000 7f442effd640 1 -- 172.21.15.72:0/2719619891 <== mon.1 v1:172.21.15.89:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f4434001a10 con 0x7f4430152c90 2024-03-29T11:35:32.596 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.595+0000 7f442effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:32.596 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.595+0000 7f442effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:32.596 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.595+0000 7f443ca4c640 4 set_mon_vals no callback set 2024-03-29T11:35:32.596 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.595+0000 7f443f251640 10 client.4255.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:32.596 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.595+0000 7f442effd640 10 monclient: dump: 2024-03-29T11:35:32.596 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:35:32.596 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:32.596 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:32.597 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:32.597 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:32.597 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:32.597 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:32.597 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:32.597 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:32.597 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:32.597 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.595+0000 7f443f251640 10 monclient: _renew_subs 2024-03-29T11:35:32.597 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.595+0000 7f443f251640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:32.597 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.595+0000 7f443f251640 1 -- 172.21.15.72:0/2719619891 --> v1:172.21.15.89:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f443016d070 con 0x7f4430152c90 2024-03-29T11:35:32.597 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.595+0000 7f443f251640 1 librados: init done 2024-03-29T11:35:32.598 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.597+0000 7f442effd640 1 -- 172.21.15.72:0/2719619891 <== mon.1 v1:172.21.15.89:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f44340039a0 con 0x7f4430152c90 2024-03-29T11:35:32.599 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.598+0000 7f443e24f640 10 monclient: get_auth_request con 0x7f441c06d940 auth_method 0 2024-03-29T11:35:32.599 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.598+0000 7f442effd640 1 -- 172.21.15.72:0/2719619891 <== mon.1 v1:172.21.15.89:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f4434083700 con 0x7f4430152c90 2024-03-29T11:35:32.599 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.598+0000 7f442effd640 10 client.4255.objecter ms_dispatch 0x7f4438000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:32.599 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.598+0000 7f442effd640 3 client.4255.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:32.599 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.598+0000 7f442effd640 3 client.4255.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:32.599 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.598+0000 7f443f251640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:32.599 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.598+0000 7f443f251640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:32.599 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.598+0000 7f443f251640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:32.599 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.598+0000 7f443f251640 1 -- 172.21.15.72:0/2719619891 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4430007b70 con 0x7f4430152c90 2024-03-29T11:35:32.599 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.599+0000 7f442effd640 20 client.4255.objecter dump_active .. 0 homeless 2024-03-29T11:35:32.599 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.599+0000 7f442effd640 10 client.4255.objecter ms_handle_connect 0x7f441c06d940 2024-03-29T11:35:32.605 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.606+0000 7f442effd640 1 -- 172.21.15.72:0/2719619891 <== mon.1 v1:172.21.15.89:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f443404efe0 con 0x7f4430152c90 2024-03-29T11:35:32.605 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.606+0000 7f442effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:32.605 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.606+0000 7f442effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:32.928 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.929+0000 7f443f251640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:32.928 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.929+0000 7f443f251640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:32.928 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.929+0000 7f443f251640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:32.928 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.929+0000 7f443f251640 1 -- 172.21.15.72:0/2719619891 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4430005120 con 0x7f4430152c90 2024-03-29T11:35:32.930 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.931+0000 7f442effd640 1 -- 172.21.15.72:0/2719619891 <== mon.1 v1:172.21.15.89:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f4434050f10 con 0x7f4430152c90 2024-03-29T11:35:32.930 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.931+0000 7f442effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:32.930 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.931+0000 7f442effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:32.932 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:35:32.932 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:32.935 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.937+0000 7f443f251640 10 librados: watch_flush enter 2024-03-29T11:35:32.935 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.937+0000 7f443f251640 10 librados: watch_flush exit 2024-03-29T11:35:32.935 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.937+0000 7f443f251640 20 client.4255.objecter shutdown clearing up homeless session... 2024-03-29T11:35:32.935 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.937+0000 7f443f251640 10 client.4255.objecter successfully canceled tick 2024-03-29T11:35:32.937 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.939+0000 7f443f251640 1 -- 172.21.15.72:0/2719619891 >> v1:172.21.15.72:6816/1578866102 conn(0x7f441c06d940 legacy=0x7f441c06fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:32.937 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.939+0000 7f443f251640 10 monclient: shutdown 2024-03-29T11:35:32.937 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.939+0000 7f443f251640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:32.937 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.939+0000 7f443f251640 1 -- 172.21.15.72:0/2719619891 >> v1:172.21.15.89:6789/0 conn(0x7f4430152c90 legacy=0x7f443016b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:32.939 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.940+0000 7f443f251640 1 -- 172.21.15.72:0/2719619891 shutdown_connections 2024-03-29T11:35:32.939 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.940+0000 7f443f251640 1 -- 172.21.15.72:0/2719619891 >> 172.21.15.72:0/2719619891 conn(0x7f4430005c00 msgr2=0x7f4430031190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:32.939 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.941+0000 7f443f251640 1 -- 172.21.15.72:0/2719619891 shutdown_connections 2024-03-29T11:35:32.939 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.941+0000 7f443f251640 1 -- 172.21.15.72:0/2719619891 wait complete. 2024-03-29T11:35:32.939 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:32.941+0000 7f443f251640 1 librados: shutdown 2024-03-29T11:35:32.960 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:39.767 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:39.918 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.920+0000 7f2ad9f13640 10 monclient: get_monmap_and_config 2024-03-29T11:35:39.918 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.920+0000 7f2ad9f13640 10 monclient: build_initial_monmap 2024-03-29T11:35:39.919 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.920+0000 7f2ad9f13640 10 monclient: monmap: 2024-03-29T11:35:39.919 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:35:39.919 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:39.919 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:35:39.921139+0000 2024-03-29T11:35:39.919 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:35:39.921139+0000 2024-03-29T11:35:39.919 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:39.919 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:39.920 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:35:39.920 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:35:39.920 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:35:39.923 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:39.925 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.921+0000 7f2ad9f13640 1 Processor -- start 2024-03-29T11:35:39.925 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.922+0000 7f2ad9f13640 1 -- start start 2024-03-29T11:35:39.925 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.922+0000 7f2ad9f13640 10 monclient: init 2024-03-29T11:35:39.925 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.923+0000 7f2ad9f13640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:39.925 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.923+0000 7f2ad9f13640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:35:39.925 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.923+0000 7f2ad9f13640 10 monclient(hunting): picked mon.noname-c con 0x7f2ad40b0860 addr v1:172.21.15.89:6789/0 2024-03-29T11:35:39.926 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.923+0000 7f2ad9f13640 10 monclient(hunting): picked mon.noname-b con 0x7f2ad413ef30 addr v1:172.21.15.72:6790/0 2024-03-29T11:35:39.926 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.923+0000 7f2ad9f13640 10 monclient(hunting): picked mon.noname-a con 0x7f2ad40af8e0 addr v1:172.21.15.72:6789/0 2024-03-29T11:35:39.926 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.923+0000 7f2ad9f13640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad40b0c40 con 0x7f2ad40af8e0 2024-03-29T11:35:39.926 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.923+0000 7f2ad9f13640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad40aed10 con 0x7f2ad40b0860 2024-03-29T11:35:39.926 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.923+0000 7f2ad9f13640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad40afd50 con 0x7f2ad413ef30 2024-03-29T11:35:39.926 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.923+0000 7f2ad9f13640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:39.926 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.923+0000 7f2ad9f13640 10 monclient(hunting): authenticate will time out at 899.211243s 2024-03-29T11:35:39.928 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.929+0000 7f2ad37fe640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f2ad40b0860 0x7f2ad413e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:34235/0 (socket says 172.21.15.72:34235) 2024-03-29T11:35:39.928 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.929+0000 7f2ad37fe640 1 -- 172.21.15.72:0/2866116055 learned_addr learned my addr 172.21.15.72:0/2866116055 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:35:39.931 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.933+0000 7f2ad27fc640 1 -- 172.21.15.72:0/2866116055 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3050664209 0 0) 0x7f2ad40aed10 con 0x7f2ad40b0860 2024-03-29T11:35:39.939 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.933+0000 7f2ad27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:39.939 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.933+0000 7f2ad27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:39.939 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.933+0000 7f2ad27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:39.939 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.933+0000 7f2ad27fc640 10 monclient(hunting): my global_id is 4258 2024-03-29T11:35:39.939 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.933+0000 7f2ad27fc640 1 -- 172.21.15.72:0/2866116055 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2abc003520 con 0x7f2ad40b0860 2024-03-29T11:35:39.939 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.934+0000 7f2ad27fc640 1 -- 172.21.15.72:0/2866116055 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3728546251 0 0) 0x7f2ad40afd50 con 0x7f2ad413ef30 2024-03-29T11:35:39.939 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.934+0000 7f2ad27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:39.940 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.934+0000 7f2ad27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:39.940 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.934+0000 7f2ad27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:39.940 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.934+0000 7f2ad27fc640 10 monclient(hunting): my global_id is 4253 2024-03-29T11:35:39.940 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.934+0000 7f2ad27fc640 1 -- 172.21.15.72:0/2866116055 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ad40aed10 con 0x7f2ad413ef30 2024-03-29T11:35:39.940 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.934+0000 7f2ad27fc640 1 -- 172.21.15.72:0/2866116055 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3941423385 0 0) 0x7f2ad40b0c40 con 0x7f2ad40af8e0 2024-03-29T11:35:39.940 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.934+0000 7f2ad27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:39.940 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.934+0000 7f2ad27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:39.940 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.934+0000 7f2ad27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:39.940 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.934+0000 7f2ad27fc640 10 monclient(hunting): my global_id is 4248 2024-03-29T11:35:39.940 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.934+0000 7f2ad27fc640 1 -- 172.21.15.72:0/2866116055 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2abc0039d0 con 0x7f2ad40af8e0 2024-03-29T11:35:39.941 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.934+0000 7f2ad27fc640 1 -- 172.21.15.72:0/2866116055 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3730826128 0 0) 0x7f2ad40aed10 con 0x7f2ad413ef30 2024-03-29T11:35:39.941 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.934+0000 7f2ad27fc640 1 -- 172.21.15.72:0/2866116055 >> v1:172.21.15.89:6789/0 conn(0x7f2ad40b0860 legacy=0x7f2ad413e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:39.941 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.934+0000 7f2ad27fc640 1 -- 172.21.15.72:0/2866116055 >> v1:172.21.15.72:6789/0 conn(0x7f2ad40af8e0 legacy=0x7f2ad4149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:39.941 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.934+0000 7f2ad27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:35:39.941 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.934+0000 7f2ad27fc640 1 monclient: found mon.noname-b 2024-03-29T11:35:39.941 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.934+0000 7f2ad27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:39.941 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.934+0000 7f2ad27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.72:6790/0 2024-03-29T11:35:39.941 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.934+0000 7f2ad27fc640 1 -- 172.21.15.72:0/2866116055 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ad40ab1b0 con 0x7f2ad413ef30 2024-03-29T11:35:39.942 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.934+0000 7f2ad27fc640 10 monclient: _finish_auth 0 2024-03-29T11:35:39.942 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.934+0000 7f2ad27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:35:39.942 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.934+0000 7f2ad27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:39.942 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.934+0000 7f2ad9f13640 5 monclient: authenticate success, global_id 4253 2024-03-29T11:35:39.942 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.934+0000 7f2ad9f13640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:39.942 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.934+0000 7f2ad27fc640 1 -- 172.21.15.72:0/2866116055 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f2acc0019b0 con 0x7f2ad413ef30 2024-03-29T11:35:39.943 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.934+0000 7f2ad27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:39.943 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.934+0000 7f2ad27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:39.943 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.934+0000 7f2ad27fc640 10 monclient: dump: 2024-03-29T11:35:39.943 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:35:39.943 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:39.943 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:39.943 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:39.944 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:39.944 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:39.944 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:39.944 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:39.944 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:39.944 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:39.945 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.935+0000 7f2ad9f13640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:39.945 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.936+0000 7f2ad27fc640 1 -- 172.21.15.72:0/2866116055 <== mon.2 v1:172.21.15.72:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2acc001c00 con 0x7f2ad413ef30 2024-03-29T11:35:39.945 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.936+0000 7f2ad27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:39.945 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.936+0000 7f2ad27fc640 1 -- 172.21.15.72:0/2866116055 <== mon.2 v1:172.21.15.72:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f2acc0019b0 con 0x7f2ad413ef30 2024-03-29T11:35:39.945 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.936+0000 7f2ad27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:39.945 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.936+0000 7f2ad27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:39.945 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.936+0000 7f2ad27fc640 10 monclient: dump: 2024-03-29T11:35:39.945 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:35:39.945 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:39.945 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:39.946 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:39.946 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:39.946 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:39.946 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:39.946 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:39.946 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:39.946 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:39.946 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.936+0000 7f2ad9f13640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:39.946 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.936+0000 7f2ad9f13640 4 set_mon_vals no callback set 2024-03-29T11:35:39.946 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.937+0000 7f2ad9f13640 10 monclient: shutdown 2024-03-29T11:35:39.946 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.937+0000 7f2ad9f13640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:39.947 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.937+0000 7f2ad9f13640 1 -- 172.21.15.72:0/2866116055 >> v1:172.21.15.72:6790/0 conn(0x7f2ad413ef30 legacy=0x7f2ad40af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:39.947 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.937+0000 7f2ad9f13640 1 -- 172.21.15.72:0/2866116055 shutdown_connections 2024-03-29T11:35:39.947 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.937+0000 7f2ad9f13640 1 -- 172.21.15.72:0/2866116055 >> 172.21.15.72:0/2866116055 conn(0x7f2ad406b660 msgr2=0x7f2ad406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:39.947 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.937+0000 7f2ad9f13640 1 -- 172.21.15.72:0/2866116055 shutdown_connections 2024-03-29T11:35:39.947 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.938+0000 7f2ad9f13640 1 -- 172.21.15.72:0/2866116055 wait complete. 2024-03-29T11:35:39.952 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.938+0000 7f2ad9f13640 10 monclient: build_initial_monmap 2024-03-29T11:35:39.953 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.938+0000 7f2ad9f13640 10 monclient: monmap: 2024-03-29T11:35:39.953 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:35:39.953 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:39.953 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:35:39.953 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:35:39.953 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:39.953 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:39.953 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:35:39.953 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:35:39.953 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:35:39.953 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:39.954 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.939+0000 7f2ad9f13640 1 librados: starting msgr at 2024-03-29T11:35:39.954 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.939+0000 7f2ad9f13640 1 librados: starting objecter 2024-03-29T11:35:39.954 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.939+0000 7f2ad9f13640 1 Processor -- start 2024-03-29T11:35:39.954 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.939+0000 7f2ad9f13640 1 -- start start 2024-03-29T11:35:39.954 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.939+0000 7f2ad9f13640 1 librados: setting wanted keys 2024-03-29T11:35:39.954 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.939+0000 7f2ad9f13640 1 librados: calling monclient init 2024-03-29T11:35:39.954 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.939+0000 7f2ad9f13640 10 monclient: init 2024-03-29T11:35:39.954 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.940+0000 7f2ad9f13640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:39.954 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.940+0000 7f2ad9f13640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:35:39.954 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.940+0000 7f2ad9f13640 10 monclient(hunting): picked mon.noname-c con 0x7f2ad40af8e0 addr v1:172.21.15.72:6790/0 2024-03-29T11:35:39.954 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.940+0000 7f2ad9f13640 10 monclient(hunting): picked mon.noname-b con 0x7f2ad40b0860 addr v1:172.21.15.89:6789/0 2024-03-29T11:35:39.955 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.940+0000 7f2ad9f13640 10 monclient(hunting): picked mon.noname-a con 0x7f2ad41db3e0 addr v1:172.21.15.72:6789/0 2024-03-29T11:35:39.955 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.940+0000 7f2ad9f13640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad40703a0 con 0x7f2ad41db3e0 2024-03-29T11:35:39.955 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.940+0000 7f2ad9f13640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad41db990 con 0x7f2ad40b0860 2024-03-29T11:35:39.955 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.940+0000 7f2ad9f13640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad406fca0 con 0x7f2ad40af8e0 2024-03-29T11:35:39.955 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.940+0000 7f2ad9f13640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:39.955 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.940+0000 7f2ad9f13640 10 monclient(hunting): authenticate will time out at 899.228442s 2024-03-29T11:35:39.955 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.944+0000 7f2ad3fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f2ad41db3e0 0x7f2ad4070f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:40487/0 (socket says 172.21.15.72:40487) 2024-03-29T11:35:39.955 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.944+0000 7f2ad3fff640 1 -- 172.21.15.72:0/1927141777 learned_addr learned my addr 172.21.15.72:0/1927141777 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:35:39.955 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.944+0000 7f2abbfff640 10 client.?.objecter ms_handle_connect 0x7f2ad41db3e0 2024-03-29T11:35:39.955 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.944+0000 7f2abbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:39.955 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.945+0000 7f2abbfff640 1 -- 172.21.15.72:0/1927141777 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3567650310 0 0) 0x7f2ad40703a0 con 0x7f2ad41db3e0 2024-03-29T11:35:39.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.945+0000 7f2abbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:39.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.945+0000 7f2abbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:39.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.945+0000 7f2abbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:39.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.945+0000 7f2abbfff640 10 monclient(hunting): my global_id is 4251 2024-03-29T11:35:39.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.945+0000 7f2abbfff640 1 -- 172.21.15.72:0/1927141777 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ac0004eb0 con 0x7f2ad41db3e0 2024-03-29T11:35:39.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.946+0000 7f2abbfff640 1 -- 172.21.15.72:0/1927141777 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3158250043 0 0) 0x7f2ac0004eb0 con 0x7f2ad41db3e0 2024-03-29T11:35:39.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.946+0000 7f2abbfff640 1 -- 172.21.15.72:0/1927141777 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ad40703a0 con 0x7f2ad41db3e0 2024-03-29T11:35:39.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.946+0000 7f2abbfff640 1 -- 172.21.15.72:0/1927141777 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f2ac4001990 con 0x7f2ad41db3e0 2024-03-29T11:35:39.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.946+0000 7f2abbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:39.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.946+0000 7f2abbfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:35:39.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.946+0000 7f2abbfff640 10 monclient(hunting): dump: 2024-03-29T11:35:39.957 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:35:39.957 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:39.957 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:39.957 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:39.957 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:39.957 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:39.957 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:39.957 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:39.957 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:39.958 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:39.958 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.946+0000 7f2abbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:39.958 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.946+0000 7f2abbfff640 1 -- 172.21.15.72:0/1927141777 <== mon.0 v1:172.21.15.72:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3971274912 0 0) 0x7f2ad40703a0 con 0x7f2ad41db3e0 2024-03-29T11:35:39.958 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.946+0000 7f2abbfff640 1 -- 172.21.15.72:0/1927141777 >> v1:172.21.15.72:6790/0 conn(0x7f2ad40af8e0 legacy=0x7f2ad41dacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:39.958 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.946+0000 7f2abbfff640 1 -- 172.21.15.72:0/1927141777 >> v1:172.21.15.89:6789/0 conn(0x7f2ad40b0860 legacy=0x7f2ad4070810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:35:39.958 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.947+0000 7f2abbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:35:39.958 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.947+0000 7f2abbfff640 1 monclient: found mon.a 2024-03-29T11:35:39.958 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.947+0000 7f2abbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:39.958 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.947+0000 7f2abbfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:35:39.958 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.947+0000 7f2abbfff640 1 -- 172.21.15.72:0/1927141777 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ad406ff80 con 0x7f2ad41db3e0 2024-03-29T11:35:39.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.947+0000 7f2abbfff640 10 monclient: _finish_auth 0 2024-03-29T11:35:39.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.947+0000 7f2abbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:35:39.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.947+0000 7f2abbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:39.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.947+0000 7f2ad9f13640 5 monclient: authenticate success, global_id 4251 2024-03-29T11:35:39.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.947+0000 7f2ad9f13640 10 monclient: _renew_subs 2024-03-29T11:35:39.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.947+0000 7f2ad9f13640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:35:39.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.947+0000 7f2ad9f13640 1 -- 172.21.15.72:0/1927141777 --> v1:172.21.15.72:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ad407aff0 con 0x7f2ad41db3e0 2024-03-29T11:35:39.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.947+0000 7f2ad9f13640 10 client.4251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:39.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.948+0000 7f2ad9f13640 10 monclient: _renew_subs 2024-03-29T11:35:39.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.948+0000 7f2ad9f13640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:35:39.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.948+0000 7f2ad9f13640 1 -- 172.21.15.72:0/1927141777 --> v1:172.21.15.72:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ad407b4d0 con 0x7f2ad41db3e0 2024-03-29T11:35:39.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.948+0000 7f2ad9f13640 1 librados: init done 2024-03-29T11:35:39.960 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.951+0000 7f2ad9f13640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:39.960 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.952+0000 7f2abbfff640 1 -- 172.21.15.72:0/1927141777 <== mon.0 v1:172.21.15.72:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ac4001af0 con 0x7f2ad41db3e0 2024-03-29T11:35:39.960 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.952+0000 7f2ad9f13640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:39.960 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.952+0000 7f2ad9f13640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:35:39.960 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.952+0000 7f2ad9f13640 1 -- 172.21.15.72:0/1927141777 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ad4075f10 con 0x7f2ad41db3e0 2024-03-29T11:35:39.960 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.952+0000 7f2abbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:39.960 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.952+0000 7f2abbfff640 1 -- 172.21.15.72:0/1927141777 <== mon.0 v1:172.21.15.72:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f2ac4003a60 con 0x7f2ad41db3e0 2024-03-29T11:35:39.960 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.952+0000 7f2abbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:39.961 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.952+0000 7f2abbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:39.961 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.952+0000 7f2abbfff640 10 monclient: dump: 2024-03-29T11:35:39.961 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:35:39.961 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:39.961 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:39.961 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:39.961 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:39.961 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:39.961 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:39.961 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:39.961 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:39.962 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:39.963 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.954+0000 7f2ad17fa640 4 set_mon_vals no callback set 2024-03-29T11:35:39.963 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.954+0000 7f2abbfff640 1 -- 172.21.15.72:0/1927141777 <== mon.0 v1:172.21.15.72:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f2ac4003c40 con 0x7f2ad41db3e0 2024-03-29T11:35:39.963 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.954+0000 7f2abbfff640 1 -- 172.21.15.72:0/1927141777 <== mon.0 v1:172.21.15.72:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f2ac4084b10 con 0x7f2ad41db3e0 2024-03-29T11:35:39.963 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.954+0000 7f2abbfff640 10 client.4251.objecter ms_dispatch 0x7f2ad4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:39.963 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.955+0000 7f2abbfff640 3 client.4251.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:39.963 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.955+0000 7f2abbfff640 3 client.4251.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:39.963 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.955+0000 7f2abbfff640 20 client.4251.objecter dump_active .. 0 homeless 2024-03-29T11:35:39.963 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.955+0000 7f2ad37fe640 10 monclient: get_auth_request con 0x7f2ac006db70 auth_method 0 2024-03-29T11:35:39.963 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.955+0000 7f2abbfff640 10 client.4251.objecter ms_handle_connect 0x7f2ac006db70 2024-03-29T11:35:39.964 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.963+0000 7f2abbfff640 1 -- 172.21.15.72:0/1927141777 <== mon.0 v1:172.21.15.72:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2ac4050450 con 0x7f2ad41db3e0 2024-03-29T11:35:39.964 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.963+0000 7f2abbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:39.964 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:39.963+0000 7f2abbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:40.277 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:40.278+0000 7f2ad9f13640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:40.277 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:40.278+0000 7f2ad9f13640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:40.277 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:40.278+0000 7f2ad9f13640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:35:40.277 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:40.279+0000 7f2ad9f13640 1 -- 172.21.15.72:0/1927141777 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ad4072330 con 0x7f2ad41db3e0 2024-03-29T11:35:40.280 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:35:40.280 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:40.280 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:40.280+0000 7f2abbfff640 1 -- 172.21.15.72:0/1927141777 <== mon.0 v1:172.21.15.72:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f2ac40841c0 con 0x7f2ad41db3e0 2024-03-29T11:35:40.280 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:40.280+0000 7f2abbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:40.280 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:40.280+0000 7f2abbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:40.283 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:40.284+0000 7f2ad9f13640 10 librados: watch_flush enter 2024-03-29T11:35:40.283 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:40.284+0000 7f2ad9f13640 10 librados: watch_flush exit 2024-03-29T11:35:40.283 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:40.284+0000 7f2ad9f13640 20 client.4251.objecter shutdown clearing up homeless session... 2024-03-29T11:35:40.283 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:40.284+0000 7f2ad9f13640 10 client.4251.objecter successfully canceled tick 2024-03-29T11:35:40.283 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:40.284+0000 7f2ad9f13640 1 -- 172.21.15.72:0/1927141777 >> v1:172.21.15.72:6816/1578866102 conn(0x7f2ac006db70 legacy=0x7f2ac0070030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:40.283 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:40.284+0000 7f2ad9f13640 10 monclient: shutdown 2024-03-29T11:35:40.283 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:40.284+0000 7f2ad9f13640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:40.283 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:40.284+0000 7f2ad9f13640 1 -- 172.21.15.72:0/1927141777 >> v1:172.21.15.72:6789/0 conn(0x7f2ad41db3e0 legacy=0x7f2ad4070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:40.284 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:40.285+0000 7f2ad9f13640 1 -- 172.21.15.72:0/1927141777 shutdown_connections 2024-03-29T11:35:40.284 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:40.285+0000 7f2ad9f13640 1 -- 172.21.15.72:0/1927141777 >> 172.21.15.72:0/1927141777 conn(0x7f2ad406b660 msgr2=0x7f2ad413d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:40.284 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:40.285+0000 7f2ad9f13640 1 -- 172.21.15.72:0/1927141777 shutdown_connections 2024-03-29T11:35:40.284 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:40.285+0000 7f2ad9f13640 1 -- 172.21.15.72:0/1927141777 wait complete. 2024-03-29T11:35:40.285 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:40.286+0000 7f2ad9f13640 1 librados: shutdown 2024-03-29T11:35:40.301 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:47.106 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:47.253 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.254+0000 7f66db663640 10 monclient: get_monmap_and_config 2024-03-29T11:35:47.254 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.254+0000 7f66db663640 10 monclient: build_initial_monmap 2024-03-29T11:35:47.254 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.254+0000 7f66db663640 10 monclient: monmap: 2024-03-29T11:35:47.254 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:35:47.254 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:47.254 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:35:47.255304+0000 2024-03-29T11:35:47.254 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:35:47.255304+0000 2024-03-29T11:35:47.254 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:47.254 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:47.254 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:35:47.254 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:35:47.254 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:35:47.254 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:47.255 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.256+0000 7f66db663640 1 Processor -- start 2024-03-29T11:35:47.256 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.256+0000 7f66db663640 1 -- start start 2024-03-29T11:35:47.256 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.256+0000 7f66db663640 10 monclient: init 2024-03-29T11:35:47.256 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.257+0000 7f66db663640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:47.256 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.257+0000 7f66db663640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:35:47.256 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.257+0000 7f66db663640 10 monclient(hunting): picked mon.noname-c con 0x7f66d40a9ac0 addr v1:172.21.15.89:6789/0 2024-03-29T11:35:47.256 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.257+0000 7f66db663640 10 monclient(hunting): picked mon.noname-b con 0x7f66d413ee90 addr v1:172.21.15.72:6790/0 2024-03-29T11:35:47.257 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.257+0000 7f66db663640 10 monclient(hunting): picked mon.noname-a con 0x7f66d40af900 addr v1:172.21.15.72:6789/0 2024-03-29T11:35:47.257 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.257+0000 7f66db663640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66d40afd10 con 0x7f66d40af900 2024-03-29T11:35:47.257 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.257+0000 7f66db663640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66d40af6f0 con 0x7f66d40a9ac0 2024-03-29T11:35:47.257 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.257+0000 7f66db663640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66d40aac80 con 0x7f66d413ee90 2024-03-29T11:35:47.257 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.257+0000 7f66db663640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:47.257 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.257+0000 7f66db663640 10 monclient(hunting): authenticate will time out at 906.545408s 2024-03-29T11:35:47.264 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.262+0000 7f66d9c00640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f66d40af900 0x7f66d4149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:41603/0 (socket says 172.21.15.72:41603) 2024-03-29T11:35:47.264 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.262+0000 7f66d9c00640 1 -- 172.21.15.72:0/2411188006 learned_addr learned my addr 172.21.15.72:0/2411188006 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:35:47.264 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.263+0000 7f66cbfff640 1 -- 172.21.15.72:0/2411188006 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1292575635 0 0) 0x7f66d40afd10 con 0x7f66d40af900 2024-03-29T11:35:47.264 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.263+0000 7f66cbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:47.264 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.263+0000 7f66cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:47.264 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.263+0000 7f66cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:47.264 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.263+0000 7f66cbfff640 10 monclient(hunting): my global_id is 4254 2024-03-29T11:35:47.264 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.263+0000 7f66cbfff640 1 -- 172.21.15.72:0/2411188006 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66bc003500 con 0x7f66d40af900 2024-03-29T11:35:47.264 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.263+0000 7f66cbfff640 1 -- 172.21.15.72:0/2411188006 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 219283543 0 0) 0x7f66d40aac80 con 0x7f66d413ee90 2024-03-29T11:35:47.265 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.263+0000 7f66cbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:47.265 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.263+0000 7f66cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:47.265 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.263+0000 7f66cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:47.265 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.263+0000 7f66cbfff640 10 monclient(hunting): my global_id is 4256 2024-03-29T11:35:47.265 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.263+0000 7f66cbfff640 1 -- 172.21.15.72:0/2411188006 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66d40afd10 con 0x7f66d413ee90 2024-03-29T11:35:47.265 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.264+0000 7f66cbfff640 1 -- 172.21.15.72:0/2411188006 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3546214894 0 0) 0x7f66d40afd10 con 0x7f66d413ee90 2024-03-29T11:35:47.265 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.264+0000 7f66cbfff640 1 -- 172.21.15.72:0/2411188006 >> v1:172.21.15.89:6789/0 conn(0x7f66d40a9ac0 legacy=0x7f66d40adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:35:47.265 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.264+0000 7f66cbfff640 1 -- 172.21.15.72:0/2411188006 >> v1:172.21.15.72:6789/0 conn(0x7f66d40af900 legacy=0x7f66d4149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:47.265 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.264+0000 7f66cbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:35:47.265 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.264+0000 7f66cbfff640 1 monclient: found mon.noname-b 2024-03-29T11:35:47.265 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.264+0000 7f66cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:47.265 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.264+0000 7f66cbfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.72:6790/0 2024-03-29T11:35:47.265 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.264+0000 7f66cbfff640 1 -- 172.21.15.72:0/2411188006 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d40aaf60 con 0x7f66d413ee90 2024-03-29T11:35:47.266 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.264+0000 7f66cbfff640 10 monclient: _finish_auth 0 2024-03-29T11:35:47.266 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.264+0000 7f66cbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:35:47.266 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.264+0000 7f66cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:47.266 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.264+0000 7f66cbfff640 1 -- 172.21.15.72:0/2411188006 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f66d0001670 con 0x7f66d413ee90 2024-03-29T11:35:47.266 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.264+0000 7f66cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:47.266 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.264+0000 7f66cbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:47.266 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.264+0000 7f66cbfff640 10 monclient: dump: 2024-03-29T11:35:47.266 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:35:47.266 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:47.266 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:47.266 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:47.266 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:47.266 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:47.266 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:47.266 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:47.267 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:47.267 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:47.267 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.264+0000 7f66cbfff640 1 -- 172.21.15.72:0/2411188006 <== mon.2 v1:172.21.15.72:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f66d00017d0 con 0x7f66d413ee90 2024-03-29T11:35:47.267 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.264+0000 7f66cbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:47.267 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.264+0000 7f66cbfff640 1 -- 172.21.15.72:0/2411188006 <== mon.2 v1:172.21.15.72:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f66d0001c20 con 0x7f66d413ee90 2024-03-29T11:35:47.267 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.264+0000 7f66cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:47.267 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.264+0000 7f66cbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:47.267 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.264+0000 7f66cbfff640 10 monclient: dump: 2024-03-29T11:35:47.267 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:35:47.267 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:47.267 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:47.267 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:47.267 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:47.267 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:47.267 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:47.268 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:47.268 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:47.268 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:47.268 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.264+0000 7f66db663640 5 monclient: authenticate success, global_id 4256 2024-03-29T11:35:47.268 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.264+0000 7f66db663640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:47.268 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.264+0000 7f66db663640 4 set_mon_vals no callback set 2024-03-29T11:35:47.268 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.265+0000 7f66db663640 10 monclient: shutdown 2024-03-29T11:35:47.268 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.265+0000 7f66db663640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:47.268 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.265+0000 7f66db663640 1 -- 172.21.15.72:0/2411188006 >> v1:172.21.15.72:6790/0 conn(0x7f66d413ee90 legacy=0x7f66d413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:47.269 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.268+0000 7f66db663640 1 -- 172.21.15.72:0/2411188006 shutdown_connections 2024-03-29T11:35:47.269 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.268+0000 7f66db663640 1 -- 172.21.15.72:0/2411188006 >> 172.21.15.72:0/2411188006 conn(0x7f66d40b35e0 msgr2=0x7f66d40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:47.269 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.271+0000 7f66db663640 1 -- 172.21.15.72:0/2411188006 shutdown_connections 2024-03-29T11:35:47.269 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.271+0000 7f66db663640 1 -- 172.21.15.72:0/2411188006 wait complete. 2024-03-29T11:35:47.270 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.272+0000 7f66db663640 10 monclient: build_initial_monmap 2024-03-29T11:35:47.270 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.272+0000 7f66db663640 10 monclient: monmap: 2024-03-29T11:35:47.270 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:35:47.270 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:47.271 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:35:47.271 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:35:47.271 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:47.271 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:47.271 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:35:47.271 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:35:47.271 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:35:47.271 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:47.272 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.272+0000 7f66db663640 1 librados: starting msgr at 2024-03-29T11:35:47.272 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.272+0000 7f66db663640 1 librados: starting objecter 2024-03-29T11:35:47.272 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.272+0000 7f66db663640 1 Processor -- start 2024-03-29T11:35:47.272 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.273+0000 7f66db663640 1 -- start start 2024-03-29T11:35:47.272 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.273+0000 7f66db663640 1 librados: setting wanted keys 2024-03-29T11:35:47.272 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.273+0000 7f66db663640 1 librados: calling monclient init 2024-03-29T11:35:47.272 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.273+0000 7f66db663640 10 monclient: init 2024-03-29T11:35:47.275 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.273+0000 7f66db663640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:47.275 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.273+0000 7f66db663640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:35:47.276 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.273+0000 7f66db663640 10 monclient(hunting): picked mon.noname-c con 0x7f66d40a9ac0 addr v1:172.21.15.72:6790/0 2024-03-29T11:35:47.276 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.273+0000 7f66db663640 10 monclient(hunting): picked mon.noname-a con 0x7f66d40af900 addr v1:172.21.15.72:6789/0 2024-03-29T11:35:47.276 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.273+0000 7f66db663640 10 monclient(hunting): picked mon.noname-b con 0x7f66d413ee90 addr v1:172.21.15.89:6789/0 2024-03-29T11:35:47.276 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.273+0000 7f66db663640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66d40b2b60 con 0x7f66d40af900 2024-03-29T11:35:47.276 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.273+0000 7f66db663640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66d41333d0 con 0x7f66d413ee90 2024-03-29T11:35:47.276 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.273+0000 7f66db663640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66d40b3060 con 0x7f66d40a9ac0 2024-03-29T11:35:47.276 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.273+0000 7f66db663640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:47.276 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.273+0000 7f66db663640 10 monclient(hunting): authenticate will time out at 906.561848s 2024-03-29T11:35:47.276 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.276+0000 7f66d9c00640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f66d413ee90 0x7f66d41326f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:39639/0 (socket says 172.21.15.72:39639) 2024-03-29T11:35:47.276 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.276+0000 7f66d9c00640 1 -- 172.21.15.72:0/3940141082 learned_addr learned my addr 172.21.15.72:0/3940141082 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:35:47.277 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.279+0000 7f66c9ffb640 10 client.?.objecter ms_handle_connect 0x7f66d413ee90 2024-03-29T11:35:47.277 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.279+0000 7f66c9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:47.278 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.279+0000 7f66c9ffb640 1 -- 172.21.15.72:0/3940141082 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2842779058 0 0) 0x7f66d41333d0 con 0x7f66d413ee90 2024-03-29T11:35:47.279 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.279+0000 7f66c9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:47.279 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.279+0000 7f66c9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:47.279 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.279+0000 7f66c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:47.279 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.279+0000 7f66c9ffb640 10 monclient(hunting): my global_id is 4261 2024-03-29T11:35:47.279 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.279+0000 7f66c9ffb640 1 -- 172.21.15.72:0/3940141082 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66c0004eb0 con 0x7f66d413ee90 2024-03-29T11:35:47.279 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.280+0000 7f66c9ffb640 1 -- 172.21.15.72:0/3940141082 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2115335852 0 0) 0x7f66c0004eb0 con 0x7f66d413ee90 2024-03-29T11:35:47.279 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.280+0000 7f66c9ffb640 1 -- 172.21.15.72:0/3940141082 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66d41333d0 con 0x7f66d413ee90 2024-03-29T11:35:47.279 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.280+0000 7f66c9ffb640 1 -- 172.21.15.72:0/3940141082 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f66cc001580 con 0x7f66d413ee90 2024-03-29T11:35:47.279 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.280+0000 7f66c9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:47.279 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.280+0000 7f66c9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:47.279 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.280+0000 7f66c9ffb640 10 monclient(hunting): dump: 2024-03-29T11:35:47.279 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:35:47.279 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:47.280 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:47.280 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:47.280 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:47.280 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:47.280 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:47.280 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:47.280 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:47.280 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:47.280 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.280+0000 7f66c9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:47.281 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.280+0000 7f66c9ffb640 10 client.?.objecter ms_handle_connect 0x7f66d40af900 2024-03-29T11:35:47.281 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.280+0000 7f66c9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:47.282 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.280+0000 7f66c9ffb640 1 -- 172.21.15.72:0/3940141082 <== mon.1 v1:172.21.15.89:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 392603730 0 0) 0x7f66d41333d0 con 0x7f66d413ee90 2024-03-29T11:35:47.282 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.281+0000 7f66c9ffb640 1 -- 172.21.15.72:0/3940141082 >> v1:172.21.15.72:6790/0 conn(0x7f66d40a9ac0 legacy=0x7f66d413f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:47.282 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.281+0000 7f66c9ffb640 1 -- 172.21.15.72:0/3940141082 >> v1:172.21.15.72:6789/0 conn(0x7f66d40af900 legacy=0x7f66d41d2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:47.282 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.281+0000 7f66c9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:35:47.282 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.281+0000 7f66c9ffb640 1 monclient: found mon.b 2024-03-29T11:35:47.282 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.281+0000 7f66c9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:47.282 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.281+0000 7f66c9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:47.282 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.281+0000 7f66c9ffb640 1 -- 172.21.15.72:0/3940141082 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d40b31f0 con 0x7f66d413ee90 2024-03-29T11:35:47.282 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.281+0000 7f66c9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:35:47.282 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.281+0000 7f66c9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:35:47.282 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.281+0000 7f66c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:47.282 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.281+0000 7f66db663640 5 monclient: authenticate success, global_id 4261 2024-03-29T11:35:47.282 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.281+0000 7f66db663640 10 monclient: _renew_subs 2024-03-29T11:35:47.282 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.281+0000 7f66db663640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:47.282 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.281+0000 7f66db663640 1 -- 172.21.15.72:0/3940141082 --> v1:172.21.15.89:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66d41d36b0 con 0x7f66d413ee90 2024-03-29T11:35:47.283 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.281+0000 7f66db663640 10 client.4261.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:47.283 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.281+0000 7f66db663640 10 monclient: _renew_subs 2024-03-29T11:35:47.283 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.281+0000 7f66db663640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:47.283 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.281+0000 7f66db663640 1 -- 172.21.15.72:0/3940141082 --> v1:172.21.15.89:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f66d4138360 con 0x7f66d413ee90 2024-03-29T11:35:47.283 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.281+0000 7f66db663640 1 librados: init done 2024-03-29T11:35:47.283 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.283+0000 7f66db663640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:47.283 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.283+0000 7f66db663640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:47.283 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.283+0000 7f66db663640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:47.283 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.283+0000 7f66db663640 1 -- 172.21.15.72:0/3940141082 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66d4141030 con 0x7f66d413ee90 2024-03-29T11:35:47.285 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.284+0000 7f66c9ffb640 1 -- 172.21.15.72:0/3940141082 <== mon.1 v1:172.21.15.89:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f66cc0016e0 con 0x7f66d413ee90 2024-03-29T11:35:47.285 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.284+0000 7f66c9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:47.285 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.284+0000 7f66c9ffb640 1 -- 172.21.15.72:0/3940141082 <== mon.1 v1:172.21.15.89:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f66cc001e10 con 0x7f66d413ee90 2024-03-29T11:35:47.285 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.284+0000 7f66c9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:47.285 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.284+0000 7f66c9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:47.285 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.284+0000 7f66c9ffb640 10 monclient: dump: 2024-03-29T11:35:47.285 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:35:47.285 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:47.286 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:47.286 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:47.286 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:47.286 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:47.286 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:47.286 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:47.286 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:47.286 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:47.286 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.285+0000 7f66c9ffb640 1 -- 172.21.15.72:0/3940141082 <== mon.1 v1:172.21.15.89:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f66cc01b0e0 con 0x7f66d413ee90 2024-03-29T11:35:47.286 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.284+0000 7f66caffd640 4 set_mon_vals no callback set 2024-03-29T11:35:47.286 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.285+0000 7f66c9ffb640 1 -- 172.21.15.72:0/3940141082 <== mon.1 v1:172.21.15.89:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f66cc084ee0 con 0x7f66d413ee90 2024-03-29T11:35:47.286 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.285+0000 7f66c9ffb640 10 client.4261.objecter ms_dispatch 0x7f66d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:47.286 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.285+0000 7f66c9ffb640 3 client.4261.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:47.286 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.285+0000 7f66c9ffb640 3 client.4261.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:47.286 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.285+0000 7f66c9ffb640 20 client.4261.objecter dump_active .. 0 homeless 2024-03-29T11:35:47.287 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.286+0000 7f66d93ff640 10 monclient: get_auth_request con 0x7f66c006db30 auth_method 0 2024-03-29T11:35:47.287 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.286+0000 7f66c9ffb640 10 client.4261.objecter ms_handle_connect 0x7f66c006db30 2024-03-29T11:35:47.289 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.291+0000 7f66c9ffb640 1 -- 172.21.15.72:0/3940141082 <== mon.1 v1:172.21.15.89:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f66cc0507c0 con 0x7f66d413ee90 2024-03-29T11:35:47.289 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.291+0000 7f66c9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:47.289 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.291+0000 7f66c9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:47.609 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.610+0000 7f66db663640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:47.609 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.610+0000 7f66db663640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:47.609 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.610+0000 7f66db663640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:47.609 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.610+0000 7f66db663640 1 -- 172.21.15.72:0/3940141082 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f66d40b2cf0 con 0x7f66d413ee90 2024-03-29T11:35:47.613 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.614+0000 7f66c9ffb640 1 -- 172.21.15.72:0/3940141082 <== mon.1 v1:172.21.15.89:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f66d40b2cf0 con 0x7f66d413ee90 2024-03-29T11:35:47.613 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.614+0000 7f66c9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:47.613 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.614+0000 7f66c9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:47.613 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:35:47.614 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:47.616 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.618+0000 7f66db663640 10 librados: watch_flush enter 2024-03-29T11:35:47.618 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.619+0000 7f66db663640 10 librados: watch_flush exit 2024-03-29T11:35:47.618 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.619+0000 7f66db663640 20 client.4261.objecter shutdown clearing up homeless session... 2024-03-29T11:35:47.619 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.619+0000 7f66db663640 10 client.4261.objecter successfully canceled tick 2024-03-29T11:35:47.619 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.620+0000 7f66db663640 1 -- 172.21.15.72:0/3940141082 >> v1:172.21.15.72:6816/1578866102 conn(0x7f66c006db30 legacy=0x7f66c006fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:47.619 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.620+0000 7f66db663640 10 monclient: shutdown 2024-03-29T11:35:47.619 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.620+0000 7f66db663640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:47.619 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.620+0000 7f66db663640 1 -- 172.21.15.72:0/3940141082 >> v1:172.21.15.89:6789/0 conn(0x7f66d413ee90 legacy=0x7f66d41326f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:47.619 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.620+0000 7f66db663640 1 -- 172.21.15.72:0/3940141082 shutdown_connections 2024-03-29T11:35:47.619 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.620+0000 7f66db663640 1 -- 172.21.15.72:0/3940141082 >> 172.21.15.72:0/3940141082 conn(0x7f66d40b35e0 msgr2=0x7f66d413b090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:47.619 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.620+0000 7f66db663640 1 -- 172.21.15.72:0/3940141082 shutdown_connections 2024-03-29T11:35:47.619 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.620+0000 7f66db663640 1 -- 172.21.15.72:0/3940141082 wait complete. 2024-03-29T11:35:47.619 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:47.620+0000 7f66db663640 1 librados: shutdown 2024-03-29T11:35:47.638 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:54.445 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:54.593 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.594+0000 7ff4b6f6c640 10 monclient: get_monmap_and_config 2024-03-29T11:35:54.593 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.594+0000 7ff4b6f6c640 10 monclient: build_initial_monmap 2024-03-29T11:35:54.593 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.595+0000 7ff4b6f6c640 10 monclient: monmap: 2024-03-29T11:35:54.593 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:35:54.593 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:54.593 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:35:54.595979+0000 2024-03-29T11:35:54.593 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:35:54.595979+0000 2024-03-29T11:35:54.593 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:54.594 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:54.594 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:35:54.594 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:35:54.594 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:35:54.594 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:54.600 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.599+0000 7ff4b6f6c640 1 Processor -- start 2024-03-29T11:35:54.600 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.599+0000 7ff4b6f6c640 1 -- start start 2024-03-29T11:35:54.600 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.599+0000 7ff4b6f6c640 10 monclient: init 2024-03-29T11:35:54.600 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.600+0000 7ff4b6f6c640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:54.600 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.600+0000 7ff4b6f6c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:35:54.600 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.600+0000 7ff4b6f6c640 10 monclient(hunting): picked mon.noname-a con 0x7ff4b00b1f60 addr v1:172.21.15.72:6789/0 2024-03-29T11:35:54.600 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.600+0000 7ff4b6f6c640 10 monclient(hunting): picked mon.noname-c con 0x7ff4b0071eb0 addr v1:172.21.15.89:6789/0 2024-03-29T11:35:54.601 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.602+0000 7ff4b5509640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7ff4b0143d10 0x7ff4b0146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:53845/0 (socket says 172.21.15.72:53845) 2024-03-29T11:35:54.601 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.602+0000 7ff4b5509640 1 -- 172.21.15.72:0/3743415673 learned_addr learned my addr 172.21.15.72:0/3743415673 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:35:54.601 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.602+0000 7ff4b6f6c640 10 monclient(hunting): picked mon.noname-b con 0x7ff4b0143d10 addr v1:172.21.15.72:6790/0 2024-03-29T11:35:54.601 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.602+0000 7ff4b6f6c640 1 -- 172.21.15.72:0/3743415673 --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b006ef80 con 0x7ff4b00b1f60 2024-03-29T11:35:54.601 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.602+0000 7ff4b6f6c640 1 -- 172.21.15.72:0/3743415673 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b006f260 con 0x7ff4b0071eb0 2024-03-29T11:35:54.601 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.602+0000 7ff4b6f6c640 1 -- 172.21.15.72:0/3743415673 --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b00b40c0 con 0x7ff4b0143d10 2024-03-29T11:35:54.601 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.603+0000 7ff4b6f6c640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:54.601 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.603+0000 7ff4b6f6c640 10 monclient(hunting): authenticate will time out at 913.891046s 2024-03-29T11:35:54.606 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.606+0000 7ff4af7fe640 1 -- 172.21.15.72:0/3743415673 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4212340947 0 0) 0x7ff4b00b40c0 con 0x7ff4b0143d10 2024-03-29T11:35:54.606 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.606+0000 7ff4af7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:54.606 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.606+0000 7ff4af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:54.606 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.606+0000 7ff4af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:54.606 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.606+0000 7ff4af7fe640 10 monclient(hunting): my global_id is 4259 2024-03-29T11:35:54.606 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.606+0000 7ff4af7fe640 1 -- 172.21.15.72:0/3743415673 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff498003520 con 0x7ff4b0143d10 2024-03-29T11:35:54.606 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.606+0000 7ff4af7fe640 1 -- 172.21.15.72:0/3743415673 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3816245589 0 0) 0x7ff4b006f260 con 0x7ff4b0071eb0 2024-03-29T11:35:54.606 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.607+0000 7ff4af7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:54.606 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.607+0000 7ff4af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:54.606 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.607+0000 7ff4af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:54.606 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.607+0000 7ff4af7fe640 10 monclient(hunting): my global_id is 4264 2024-03-29T11:35:54.606 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.607+0000 7ff4af7fe640 1 -- 172.21.15.72:0/3743415673 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4b00b40c0 con 0x7ff4b0071eb0 2024-03-29T11:35:54.607 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.607+0000 7ff4af7fe640 1 -- 172.21.15.72:0/3743415673 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 655703901 0 0) 0x7ff498003520 con 0x7ff4b0143d10 2024-03-29T11:35:54.607 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.607+0000 7ff4af7fe640 1 -- 172.21.15.72:0/3743415673 >> v1:172.21.15.89:6789/0 conn(0x7ff4b0071eb0 legacy=0x7ff4b0146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:54.607 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.607+0000 7ff4af7fe640 1 -- 172.21.15.72:0/3743415673 >> v1:172.21.15.72:6789/0 conn(0x7ff4b00b1f60 legacy=0x7ff4b0143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:35:54.607 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.607+0000 7ff4af7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:35:54.607 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.607+0000 7ff4af7fe640 1 monclient: found mon.noname-b 2024-03-29T11:35:54.607 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.607+0000 7ff4af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:54.607 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.607+0000 7ff4af7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.72:6790/0 2024-03-29T11:35:54.607 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.607+0000 7ff4af7fe640 1 -- 172.21.15.72:0/3743415673 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4b0154d60 con 0x7ff4b0143d10 2024-03-29T11:35:54.607 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.607+0000 7ff4af7fe640 10 monclient: _finish_auth 0 2024-03-29T11:35:54.607 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.607+0000 7ff4af7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:35:54.607 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.607+0000 7ff4af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:54.607 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.607+0000 7ff4af7fe640 1 -- 172.21.15.72:0/3743415673 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7ff4a8001a10 con 0x7ff4b0143d10 2024-03-29T11:35:54.607 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.607+0000 7ff4af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:54.607 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.607+0000 7ff4af7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:54.607 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.607+0000 7ff4af7fe640 10 monclient: dump: 2024-03-29T11:35:54.607 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:35:54.608 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:54.608 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:54.608 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:54.608 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:54.608 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:54.608 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:54.608 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:54.608 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:54.608 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:54.608 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.607+0000 7ff4b6f6c640 5 monclient: authenticate success, global_id 4259 2024-03-29T11:35:54.608 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.607+0000 7ff4b6f6c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:54.608 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.607+0000 7ff4af7fe640 1 -- 172.21.15.72:0/3743415673 <== mon.2 v1:172.21.15.72:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4a8001c30 con 0x7ff4b0143d10 2024-03-29T11:35:54.608 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.607+0000 7ff4af7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:54.608 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.607+0000 7ff4af7fe640 1 -- 172.21.15.72:0/3743415673 <== mon.2 v1:172.21.15.72:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7ff4a8001fc0 con 0x7ff4b0143d10 2024-03-29T11:35:54.609 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.607+0000 7ff4af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:54.609 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.607+0000 7ff4af7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:54.609 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.607+0000 7ff4af7fe640 10 monclient: dump: 2024-03-29T11:35:54.609 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:35:54.609 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:54.609 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:54.609 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:54.609 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:54.609 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:54.609 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:54.609 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:54.609 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:54.609 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:54.609 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.607+0000 7ff4b6f6c640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:54.609 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.607+0000 7ff4b6f6c640 4 set_mon_vals no callback set 2024-03-29T11:35:54.610 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.608+0000 7ff4b6f6c640 10 monclient: shutdown 2024-03-29T11:35:54.611 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.608+0000 7ff4b6f6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:54.611 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.608+0000 7ff4b6f6c640 1 -- 172.21.15.72:0/3743415673 >> v1:172.21.15.72:6790/0 conn(0x7ff4b0143d10 legacy=0x7ff4b0146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:54.611 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.608+0000 7ff4b6f6c640 1 -- 172.21.15.72:0/3743415673 shutdown_connections 2024-03-29T11:35:54.611 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.608+0000 7ff4b6f6c640 1 -- 172.21.15.72:0/3743415673 >> 172.21.15.72:0/3743415673 conn(0x7ff4b0066a00 msgr2=0x7ff4b0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:54.611 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.611+0000 7ff4b6f6c640 1 -- 172.21.15.72:0/3743415673 shutdown_connections 2024-03-29T11:35:54.611 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.611+0000 7ff4b6f6c640 1 -- 172.21.15.72:0/3743415673 wait complete. 2024-03-29T11:35:54.616 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.612+0000 7ff4b6f6c640 10 monclient: build_initial_monmap 2024-03-29T11:35:54.616 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.612+0000 7ff4b6f6c640 10 monclient: monmap: 2024-03-29T11:35:54.616 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:35:54.616 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:54.616 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:35:54.616 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:35:54.616 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:54.616 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:54.617 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:35:54.617 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:35:54.617 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:35:54.617 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:54.617 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.613+0000 7ff4b6f6c640 1 librados: starting msgr at 2024-03-29T11:35:54.617 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.613+0000 7ff4b6f6c640 1 librados: starting objecter 2024-03-29T11:35:54.617 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.613+0000 7ff4b6f6c640 1 Processor -- start 2024-03-29T11:35:54.617 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.613+0000 7ff4b6f6c640 1 -- start start 2024-03-29T11:35:54.617 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.613+0000 7ff4b6f6c640 1 librados: setting wanted keys 2024-03-29T11:35:54.617 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.613+0000 7ff4b6f6c640 1 librados: calling monclient init 2024-03-29T11:35:54.617 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.613+0000 7ff4b6f6c640 10 monclient: init 2024-03-29T11:35:54.617 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.614+0000 7ff4b6f6c640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:54.617 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.614+0000 7ff4b6f6c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:35:54.617 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.614+0000 7ff4b6f6c640 10 monclient(hunting): picked mon.noname-a con 0x7ff4b00b1f60 addr v1:172.21.15.72:6789/0 2024-03-29T11:35:54.617 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.614+0000 7ff4b6f6c640 10 monclient(hunting): picked mon.noname-b con 0x7ff4b0143d10 addr v1:172.21.15.89:6789/0 2024-03-29T11:35:54.618 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.618+0000 7ff4b6f6c640 10 monclient(hunting): picked mon.noname-c con 0x7ff4b0070760 addr v1:172.21.15.72:6790/0 2024-03-29T11:35:54.618 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.618+0000 7ff4b6f6c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b006f950 con 0x7ff4b00b1f60 2024-03-29T11:35:54.618 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.618+0000 7ff4b6f6c640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b006fc30 con 0x7ff4b0143d10 2024-03-29T11:35:54.618 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.618+0000 7ff4b6f6c640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b0141770 con 0x7ff4b0070760 2024-03-29T11:35:54.618 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.618+0000 7ff4b6f6c640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:54.619 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.618+0000 7ff4b6f6c640 10 monclient(hunting): authenticate will time out at 913.906813s 2024-03-29T11:35:54.619 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.620+0000 7ff4b4d08640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7ff4b00b1f60 0x7ff4b01351d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45571/0 (socket says 172.21.15.72:45571) 2024-03-29T11:35:54.619 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.620+0000 7ff4b4d08640 1 -- 172.21.15.72:0/1125478469 learned_addr learned my addr 172.21.15.72:0/1125478469 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:35:54.619 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.621+0000 7ff4ad7fa640 10 client.?.objecter ms_handle_connect 0x7ff4b00b1f60 2024-03-29T11:35:54.619 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.621+0000 7ff4ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:54.619 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.621+0000 7ff4ad7fa640 10 client.?.objecter ms_handle_connect 0x7ff4b0143d10 2024-03-29T11:35:54.619 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.621+0000 7ff4ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:54.620 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.622+0000 7ff4ad7fa640 1 -- 172.21.15.72:0/1125478469 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2177146168 0 0) 0x7ff4b006fc30 con 0x7ff4b0143d10 2024-03-29T11:35:54.620 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.622+0000 7ff4ad7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:35:54.620 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.622+0000 7ff4ad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:54.620 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.622+0000 7ff4ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:54.620 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.622+0000 7ff4ad7fa640 10 monclient(hunting): my global_id is 4267 2024-03-29T11:35:54.620 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.622+0000 7ff4ad7fa640 1 -- 172.21.15.72:0/1125478469 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff49c004eb0 con 0x7ff4b0143d10 2024-03-29T11:35:54.621 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.622+0000 7ff4ad7fa640 1 -- 172.21.15.72:0/1125478469 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2755141730 0 0) 0x7ff49c004eb0 con 0x7ff4b0143d10 2024-03-29T11:35:54.621 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.622+0000 7ff4ad7fa640 1 -- 172.21.15.72:0/1125478469 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4b006fc30 con 0x7ff4b0143d10 2024-03-29T11:35:54.621 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.622+0000 7ff4ad7fa640 1 -- 172.21.15.72:0/1125478469 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7ff4a4001820 con 0x7ff4b0143d10 2024-03-29T11:35:54.621 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.622+0000 7ff4ad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:54.621 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.622+0000 7ff4ad7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:54.621 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.622+0000 7ff4ad7fa640 10 monclient(hunting): dump: 2024-03-29T11:35:54.621 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:35:54.621 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:54.621 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:54.621 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:54.621 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:54.621 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:54.622 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:54.622 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:54.622 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:54.622 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:54.622 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.622+0000 7ff4ad7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:54.622 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.623+0000 7ff4ad7fa640 1 -- 172.21.15.72:0/1125478469 <== mon.1 v1:172.21.15.89:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4283278673 0 0) 0x7ff4b006fc30 con 0x7ff4b0143d10 2024-03-29T11:35:54.622 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.623+0000 7ff4ad7fa640 1 -- 172.21.15.72:0/1125478469 >> v1:172.21.15.72:6790/0 conn(0x7ff4b0070760 legacy=0x7ff4b01eed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:54.622 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.623+0000 7ff4ad7fa640 1 -- 172.21.15.72:0/1125478469 >> v1:172.21.15.72:6789/0 conn(0x7ff4b00b1f60 legacy=0x7ff4b01351d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:54.622 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.623+0000 7ff4ad7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:35:54.622 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.623+0000 7ff4ad7fa640 1 monclient: found mon.b 2024-03-29T11:35:54.623 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.623+0000 7ff4ad7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:54.623 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.623+0000 7ff4ad7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:54.623 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.623+0000 7ff4ad7fa640 1 -- 172.21.15.72:0/1125478469 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4b0141a50 con 0x7ff4b0143d10 2024-03-29T11:35:54.623 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.623+0000 7ff4ad7fa640 10 monclient: _finish_auth 0 2024-03-29T11:35:54.623 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.623+0000 7ff4ad7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:35:54.623 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.623+0000 7ff4ad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:54.623 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.623+0000 7ff4b6f6c640 5 monclient: authenticate success, global_id 4267 2024-03-29T11:35:54.623 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.623+0000 7ff4b6f6c640 10 monclient: _renew_subs 2024-03-29T11:35:54.623 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.623+0000 7ff4b6f6c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:54.623 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.623+0000 7ff4b6f6c640 1 -- 172.21.15.72:0/1125478469 --> v1:172.21.15.89:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4b00716e0 con 0x7ff4b0143d10 2024-03-29T11:35:54.623 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.623+0000 7ff4b6f6c640 10 client.4267.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:54.623 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.623+0000 7ff4b6f6c640 10 monclient: _renew_subs 2024-03-29T11:35:54.623 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.623+0000 7ff4b6f6c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:54.623 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.623+0000 7ff4b6f6c640 1 -- 172.21.15.72:0/1125478469 --> v1:172.21.15.89:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4b0070bb0 con 0x7ff4b0143d10 2024-03-29T11:35:54.624 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.623+0000 7ff4b6f6c640 1 librados: init done 2024-03-29T11:35:54.624 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.623+0000 7ff4ad7fa640 1 -- 172.21.15.72:0/1125478469 <== mon.1 v1:172.21.15.89:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4a4003610 con 0x7ff4b0143d10 2024-03-29T11:35:54.624 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.623+0000 7ff4ad7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:54.624 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.623+0000 7ff4ad7fa640 1 -- 172.21.15.72:0/1125478469 <== mon.1 v1:172.21.15.89:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7ff4a4001980 con 0x7ff4b0143d10 2024-03-29T11:35:54.624 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.623+0000 7ff4ad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:54.624 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.623+0000 7ff4ad7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:54.624 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.623+0000 7ff4ad7fa640 10 monclient: dump: 2024-03-29T11:35:54.624 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:35:54.624 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:35:54.624 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:54.624 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:35:54.624 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:54.624 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:35:54.624 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:35:54.624 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:35:54.625 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:35:54.625 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:35:54.625 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.623+0000 7ff4ae7fc640 4 set_mon_vals no callback set 2024-03-29T11:35:54.626 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.626+0000 7ff4ad7fa640 1 -- 172.21.15.72:0/1125478469 <== mon.1 v1:172.21.15.89:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7ff4a4003920 con 0x7ff4b0143d10 2024-03-29T11:35:54.626 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.626+0000 7ff4ad7fa640 1 -- 172.21.15.72:0/1125478469 <== mon.1 v1:172.21.15.89:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7ff4a40837c0 con 0x7ff4b0143d10 2024-03-29T11:35:54.626 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.626+0000 7ff4ad7fa640 10 client.4267.objecter ms_dispatch 0x7ff4b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:54.626 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.626+0000 7ff4ad7fa640 3 client.4267.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:54.626 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.626+0000 7ff4ad7fa640 3 client.4267.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:54.626 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.626+0000 7ff4ad7fa640 20 client.4267.objecter dump_active .. 0 homeless 2024-03-29T11:35:54.626 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.627+0000 7ff4b4d08640 10 monclient: get_auth_request con 0x7ff49c06dc20 auth_method 0 2024-03-29T11:35:54.626 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.627+0000 7ff4ad7fa640 10 client.4267.objecter ms_handle_connect 0x7ff49c06dc20 2024-03-29T11:35:54.627 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.628+0000 7ff4b6f6c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:54.627 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.628+0000 7ff4b6f6c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:54.627 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.628+0000 7ff4b6f6c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:54.627 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.628+0000 7ff4b6f6c640 1 -- 172.21.15.72:0/1125478469 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff490005080 con 0x7ff4b0143d10 2024-03-29T11:35:54.635 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.637+0000 7ff4ad7fa640 1 -- 172.21.15.72:0/1125478469 <== mon.1 v1:172.21.15.89:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff4a404eed0 con 0x7ff4b0143d10 2024-03-29T11:35:54.635 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.637+0000 7ff4ad7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:54.635 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.637+0000 7ff4ad7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:54.961 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.963+0000 7ff4b6f6c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:54.961 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.963+0000 7ff4b6f6c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:54.961 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.963+0000 7ff4b6f6c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:35:54.962 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.963+0000 7ff4b6f6c640 1 -- 172.21.15.72:0/1125478469 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff490005330 con 0x7ff4b0143d10 2024-03-29T11:35:54.962 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.963+0000 7ff4ad7fa640 1 -- 172.21.15.72:0/1125478469 <== mon.1 v1:172.21.15.89:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7ff4a4050fd0 con 0x7ff4b0143d10 2024-03-29T11:35:54.962 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.964+0000 7ff4ad7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:54.962 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.964+0000 7ff4ad7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:54.966 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:35:54.966 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:54.968 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.970+0000 7ff4b6f6c640 10 librados: watch_flush enter 2024-03-29T11:35:54.968 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.970+0000 7ff4b6f6c640 10 librados: watch_flush exit 2024-03-29T11:35:54.968 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.970+0000 7ff4b6f6c640 20 client.4267.objecter shutdown clearing up homeless session... 2024-03-29T11:35:54.968 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.970+0000 7ff4b6f6c640 10 client.4267.objecter successfully canceled tick 2024-03-29T11:35:54.971 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.973+0000 7ff4b6f6c640 1 -- 172.21.15.72:0/1125478469 >> v1:172.21.15.72:6816/1578866102 conn(0x7ff49c06dc20 legacy=0x7ff49c0700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:54.971 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.973+0000 7ff4b6f6c640 10 monclient: shutdown 2024-03-29T11:35:54.971 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.973+0000 7ff4b6f6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:54.971 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.973+0000 7ff4b6f6c640 1 -- 172.21.15.72:0/1125478469 >> v1:172.21.15.89:6789/0 conn(0x7ff4b0143d10 legacy=0x7ff4b0070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:54.974 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.976+0000 7ff4b6f6c640 1 -- 172.21.15.72:0/1125478469 shutdown_connections 2024-03-29T11:35:54.974 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.976+0000 7ff4b6f6c640 1 -- 172.21.15.72:0/1125478469 >> 172.21.15.72:0/1125478469 conn(0x7ff4b0066a00 msgr2=0x7ff4b0146960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:54.975 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.976+0000 7ff4b6f6c640 1 -- 172.21.15.72:0/1125478469 shutdown_connections 2024-03-29T11:35:54.975 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.976+0000 7ff4b6f6c640 1 -- 172.21.15.72:0/1125478469 wait complete. 2024-03-29T11:35:54.975 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:35:54.976+0000 7ff4b6f6c640 1 librados: shutdown 2024-03-29T11:35:54.994 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:01.801 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:01.953 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.954+0000 7f276914d640 10 monclient: get_monmap_and_config 2024-03-29T11:36:01.953 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.954+0000 7f276914d640 10 monclient: build_initial_monmap 2024-03-29T11:36:01.953 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.954+0000 7f276914d640 10 monclient: monmap: 2024-03-29T11:36:01.953 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:36:01.953 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:36:01.954 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:36:01.955700+0000 2024-03-29T11:36:01.954 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:36:01.955700+0000 2024-03-29T11:36:01.954 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:01.954 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:01.954 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:36:01.954 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:36:01.954 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:36:01.954 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:01.958 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.957+0000 7f276914d640 1 Processor -- start 2024-03-29T11:36:01.958 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.957+0000 7f276914d640 1 -- start start 2024-03-29T11:36:01.958 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.957+0000 7f276914d640 10 monclient: init 2024-03-29T11:36:01.958 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.958+0000 7f276914d640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:01.958 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.958+0000 7f276914d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:36:01.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.958+0000 7f276914d640 10 monclient(hunting): picked mon.noname-c con 0x7f275c006e00 addr v1:172.21.15.89:6789/0 2024-03-29T11:36:01.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.958+0000 7f276914d640 10 monclient(hunting): picked mon.noname-a con 0x7f275c035dd0 addr v1:172.21.15.72:6789/0 2024-03-29T11:36:01.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.958+0000 7f276914d640 10 monclient(hunting): picked mon.noname-b con 0x7f275c0c9640 addr v1:172.21.15.72:6790/0 2024-03-29T11:36:01.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.958+0000 7f276914d640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f275c035940 con 0x7f275c035dd0 2024-03-29T11:36:01.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.958+0000 7f276914d640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f275c036250 con 0x7f275c006e00 2024-03-29T11:36:01.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.958+0000 7f276914d640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f275c035610 con 0x7f275c0c9640 2024-03-29T11:36:01.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.958+0000 7f276914d640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:01.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.958+0000 7f276914d640 10 monclient(hunting): authenticate will time out at 921.246839s 2024-03-29T11:36:01.961 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.962+0000 7f276359e640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f275c0c9640 0x7f275c0d7530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:48479/0 (socket says 172.21.15.72:48479) 2024-03-29T11:36:01.961 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.962+0000 7f276359e640 1 -- 172.21.15.72:0/747407861 learned_addr learned my addr 172.21.15.72:0/747407861 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:36:01.965 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.964+0000 7f2761d9b640 1 -- 172.21.15.72:0/747407861 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1893118993 0 0) 0x7f275c035940 con 0x7f275c035dd0 2024-03-29T11:36:01.965 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.964+0000 7f2761d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:36:01.965 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.964+0000 7f2761d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:01.966 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.964+0000 7f2761d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:01.966 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.964+0000 7f2761d9b640 10 monclient(hunting): my global_id is 4263 2024-03-29T11:36:01.966 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.964+0000 7f2761d9b640 1 -- 172.21.15.72:0/747407861 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f274c003520 con 0x7f275c035dd0 2024-03-29T11:36:01.966 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.964+0000 7f2761d9b640 1 -- 172.21.15.72:0/747407861 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4122423155 0 0) 0x7f274c003520 con 0x7f275c035dd0 2024-03-29T11:36:01.966 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.964+0000 7f2761d9b640 1 -- 172.21.15.72:0/747407861 >> v1:172.21.15.72:6790/0 conn(0x7f275c0c9640 legacy=0x7f275c0d7530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:01.966 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.964+0000 7f2761d9b640 1 -- 172.21.15.72:0/747407861 >> v1:172.21.15.89:6789/0 conn(0x7f275c006e00 legacy=0x7f275c0cbc90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:36:01.966 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.964+0000 7f2761d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:36:01.966 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.964+0000 7f2761d9b640 1 monclient: found mon.noname-a 2024-03-29T11:36:01.966 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.964+0000 7f2761d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:01.966 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.964+0000 7f2761d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.72:6789/0 2024-03-29T11:36:01.966 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.964+0000 7f2761d9b640 1 -- 172.21.15.72:0/747407861 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f275c034df0 con 0x7f275c035dd0 2024-03-29T11:36:01.966 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.964+0000 7f2761d9b640 10 monclient: _finish_auth 0 2024-03-29T11:36:01.966 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.964+0000 7f2761d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:36:01.966 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.964+0000 7f2761d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:01.967 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.964+0000 7f2761d9b640 1 -- 172.21.15.72:0/747407861 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f276404f730 con 0x7f275c035dd0 2024-03-29T11:36:01.967 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.964+0000 7f2761d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:01.967 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.965+0000 7f2761d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:01.967 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.965+0000 7f2761d9b640 10 monclient: dump: 2024-03-29T11:36:01.967 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:36:01.967 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:36:01.967 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:01.967 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:01.967 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:01.967 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:01.967 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:36:01.967 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:36:01.967 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:36:01.967 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:01.967 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.965+0000 7f2761d9b640 1 -- 172.21.15.72:0/747407861 <== mon.0 v1:172.21.15.72:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2764075f10 con 0x7f275c035dd0 2024-03-29T11:36:01.968 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.965+0000 7f2761d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:01.968 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.965+0000 7f2761d9b640 1 -- 172.21.15.72:0/747407861 <== mon.0 v1:172.21.15.72:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f27640758b0 con 0x7f275c035dd0 2024-03-29T11:36:01.968 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.965+0000 7f2761d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:01.968 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.965+0000 7f2761d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:01.968 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.965+0000 7f2761d9b640 10 monclient: dump: 2024-03-29T11:36:01.968 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:36:01.968 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:36:01.968 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:01.968 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:01.968 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:01.968 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:01.968 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:36:01.968 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:36:01.968 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:36:01.968 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:01.968 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.966+0000 7f276914d640 5 monclient: authenticate success, global_id 4263 2024-03-29T11:36:01.969 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.966+0000 7f276914d640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:01.969 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.966+0000 7f276914d640 4 set_mon_vals no callback set 2024-03-29T11:36:01.969 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.968+0000 7f276914d640 10 monclient: shutdown 2024-03-29T11:36:01.969 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.968+0000 7f276914d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:01.969 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.968+0000 7f276914d640 1 -- 172.21.15.72:0/747407861 >> v1:172.21.15.72:6789/0 conn(0x7f275c035dd0 legacy=0x7f275c0c8f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:01.970 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.970+0000 7f276914d640 1 -- 172.21.15.72:0/747407861 shutdown_connections 2024-03-29T11:36:01.970 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.970+0000 7f276914d640 1 -- 172.21.15.72:0/747407861 >> 172.21.15.72:0/747407861 conn(0x7f275c005bd0 msgr2=0x7f275c005fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:01.972 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.972+0000 7f276914d640 1 -- 172.21.15.72:0/747407861 shutdown_connections 2024-03-29T11:36:01.974 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.975+0000 7f276914d640 1 -- 172.21.15.72:0/747407861 wait complete. 2024-03-29T11:36:01.975 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.977+0000 7f276914d640 10 monclient: build_initial_monmap 2024-03-29T11:36:01.975 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.977+0000 7f276914d640 10 monclient: monmap: 2024-03-29T11:36:01.976 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:36:01.976 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:01.976 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:36:01.976 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:36:01.976 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:01.976 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:01.976 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:36:01.976 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:36:01.976 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:36:01.976 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:01.976 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.978+0000 7f276914d640 1 librados: starting msgr at 2024-03-29T11:36:01.979 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.981+0000 7f276914d640 1 librados: starting objecter 2024-03-29T11:36:01.980 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.982+0000 7f276914d640 1 Processor -- start 2024-03-29T11:36:01.981 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.982+0000 7f276914d640 1 -- start start 2024-03-29T11:36:01.981 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.982+0000 7f276914d640 1 librados: setting wanted keys 2024-03-29T11:36:01.981 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.982+0000 7f276914d640 1 librados: calling monclient init 2024-03-29T11:36:01.981 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.982+0000 7f276914d640 10 monclient: init 2024-03-29T11:36:01.981 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.983+0000 7f276914d640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:01.982 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.983+0000 7f276914d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:36:01.984 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.983+0000 7f276914d640 10 monclient(hunting): picked mon.noname-c con 0x7f275c006e00 addr v1:172.21.15.72:6790/0 2024-03-29T11:36:01.984 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.984+0000 7f276914d640 10 monclient(hunting): picked mon.noname-a con 0x7f275c035dd0 addr v1:172.21.15.72:6789/0 2024-03-29T11:36:01.984 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.984+0000 7f276914d640 10 monclient(hunting): picked mon.noname-b con 0x7f275c0c9640 addr v1:172.21.15.89:6789/0 2024-03-29T11:36:01.984 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.984+0000 7f276914d640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f275c02d930 con 0x7f275c035dd0 2024-03-29T11:36:01.984 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.984+0000 7f276914d640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f275c0320e0 con 0x7f275c0c9640 2024-03-29T11:36:01.984 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.984+0000 7f276914d640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f275c033140 con 0x7f275c006e00 2024-03-29T11:36:01.984 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.984+0000 7f276914d640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:01.984 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.985+0000 7f276914d640 10 monclient(hunting): authenticate will time out at 921.273009s 2024-03-29T11:36:01.985 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.986+0000 7f2762d9d640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f275c006e00 0x7f275c0328c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:48583/0 (socket says 172.21.15.72:48583) 2024-03-29T11:36:01.985 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.986+0000 7f2762d9d640 1 -- 172.21.15.72:0/3004618437 learned_addr learned my addr 172.21.15.72:0/3004618437 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:36:01.989 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.988+0000 7f274b7fe640 10 client.?.objecter ms_handle_connect 0x7f275c0c9640 2024-03-29T11:36:01.989 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.988+0000 7f274b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:01.989 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.990+0000 7f274b7fe640 1 -- 172.21.15.72:0/3004618437 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2654590009 0 0) 0x7f275c0320e0 con 0x7f275c0c9640 2024-03-29T11:36:01.989 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.990+0000 7f274b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:01.989 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.990+0000 7f274b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:01.989 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.990+0000 7f274b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:01.989 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.990+0000 7f274b7fe640 10 monclient(hunting): my global_id is 4270 2024-03-29T11:36:01.989 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.990+0000 7f274b7fe640 1 -- 172.21.15.72:0/3004618437 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2750004eb0 con 0x7f275c0c9640 2024-03-29T11:36:01.989 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.990+0000 7f274b7fe640 10 client.?.objecter ms_handle_connect 0x7f275c006e00 2024-03-29T11:36:01.990 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.990+0000 7f274b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:01.990 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.990+0000 7f274b7fe640 1 -- 172.21.15.72:0/3004618437 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1919954717 0 0) 0x7f275c033140 con 0x7f275c006e00 2024-03-29T11:36:01.990 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.990+0000 7f274b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:01.990 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.990+0000 7f274b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:01.990 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.990+0000 7f274b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:01.990 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.990+0000 7f274b7fe640 10 monclient(hunting): my global_id is 4262 2024-03-29T11:36:01.990 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.990+0000 7f274b7fe640 1 -- 172.21.15.72:0/3004618437 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27500051d0 con 0x7f275c006e00 2024-03-29T11:36:01.990 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.990+0000 7f274b7fe640 1 -- 172.21.15.72:0/3004618437 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 819255277 0 0) 0x7f27500051d0 con 0x7f275c006e00 2024-03-29T11:36:01.990 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.990+0000 7f274b7fe640 1 -- 172.21.15.72:0/3004618437 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2750005640 con 0x7f275c006e00 2024-03-29T11:36:01.990 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.990+0000 7f274b7fe640 1 -- 172.21.15.72:0/3004618437 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f27580017f0 con 0x7f275c006e00 2024-03-29T11:36:01.990 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.990+0000 7f274b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:01.991 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.990+0000 7f274b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:01.991 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.990+0000 7f274b7fe640 10 monclient(hunting): dump: 2024-03-29T11:36:01.991 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:36:01.991 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:36:01.991 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:01.991 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:01.991 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:01.991 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:01.991 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:36:01.991 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:36:01.991 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:36:01.991 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:01.992 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.990+0000 7f274b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:01.992 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.990+0000 7f274b7fe640 10 client.?.objecter ms_handle_connect 0x7f275c035dd0 2024-03-29T11:36:01.992 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.990+0000 7f274b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:01.995 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.991+0000 7f274b7fe640 1 -- 172.21.15.72:0/3004618437 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4027800900 0 0) 0x7f275c02d930 con 0x7f275c035dd0 2024-03-29T11:36:01.995 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.991+0000 7f274b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:01.995 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.991+0000 7f274b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:01.995 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.991+0000 7f274b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:01.995 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.991+0000 7f274b7fe640 10 monclient(hunting): my global_id is 4266 2024-03-29T11:36:01.995 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.991+0000 7f274b7fe640 1 -- 172.21.15.72:0/3004618437 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27500051d0 con 0x7f275c035dd0 2024-03-29T11:36:01.995 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.991+0000 7f274b7fe640 1 -- 172.21.15.72:0/3004618437 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1822543943 0 0) 0x7f2750004eb0 con 0x7f275c0c9640 2024-03-29T11:36:01.995 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.991+0000 7f274b7fe640 1 -- 172.21.15.72:0/3004618437 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2750005ea0 con 0x7f275c0c9640 2024-03-29T11:36:01.996 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.991+0000 7f274b7fe640 1 -- 172.21.15.72:0/3004618437 <== mon.2 v1:172.21.15.72:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2155158738 0 0) 0x7f2750005640 con 0x7f275c006e00 2024-03-29T11:36:01.996 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.991+0000 7f274b7fe640 1 -- 172.21.15.72:0/3004618437 >> v1:172.21.15.89:6789/0 conn(0x7f275c0c9640 legacy=0x7f275c160d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:01.996 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.991+0000 7f274b7fe640 1 -- 172.21.15.72:0/3004618437 >> v1:172.21.15.72:6789/0 conn(0x7f275c035dd0 legacy=0x7f275c0c3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:01.996 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.991+0000 7f274b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:36:01.996 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.991+0000 7f274b7fe640 1 monclient: found mon.c 2024-03-29T11:36:01.996 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.991+0000 7f274b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:01.996 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.991+0000 7f274b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:36:01.996 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.991+0000 7f274b7fe640 1 -- 172.21.15.72:0/3004618437 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f275c0c40a0 con 0x7f275c006e00 2024-03-29T11:36:01.996 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.991+0000 7f274b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:36:01.996 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.991+0000 7f274b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:36:01.996 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.991+0000 7f274b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:01.997 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.992+0000 7f276914d640 5 monclient: authenticate success, global_id 4262 2024-03-29T11:36:01.997 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.992+0000 7f276914d640 10 monclient: _renew_subs 2024-03-29T11:36:01.997 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.992+0000 7f276914d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:36:01.997 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.992+0000 7f276914d640 1 -- 172.21.15.72:0/3004618437 --> v1:172.21.15.72:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f275c0c4350 con 0x7f275c006e00 2024-03-29T11:36:01.997 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.992+0000 7f276914d640 10 client.4262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:01.997 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.992+0000 7f276914d640 10 monclient: _renew_subs 2024-03-29T11:36:01.997 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.992+0000 7f276914d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:36:01.997 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.992+0000 7f276914d640 1 -- 172.21.15.72:0/3004618437 --> v1:172.21.15.72:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f275c038740 con 0x7f275c006e00 2024-03-29T11:36:01.997 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.992+0000 7f276914d640 1 librados: init done 2024-03-29T11:36:01.997 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.995+0000 7f27497fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:01.997 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.995+0000 7f274b7fe640 1 -- 172.21.15.72:0/3004618437 <== mon.2 v1:172.21.15.72:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27580019d0 con 0x7f275c006e00 2024-03-29T11:36:01.997 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.995+0000 7f274b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:01.998 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.995+0000 7f274b7fe640 1 -- 172.21.15.72:0/3004618437 <== mon.2 v1:172.21.15.72:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f2758004aa0 con 0x7f275c006e00 2024-03-29T11:36:01.998 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.995+0000 7f27497fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:01.998 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.995+0000 7f27497fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:36:01.998 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.995+0000 7f2760d99640 4 set_mon_vals no callback set 2024-03-29T11:36:01.998 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.995+0000 7f27497fa640 1 -- 172.21.15.72:0/3004618437 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f275c007b60 con 0x7f275c006e00 2024-03-29T11:36:01.998 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.995+0000 7f274b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:01.998 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.995+0000 7f274b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:01.998 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.995+0000 7f274b7fe640 10 monclient: dump: 2024-03-29T11:36:01.998 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:36:01.998 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:36:01.998 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:01.998 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:01.998 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:01.998 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:01.999 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:36:01.999 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:36:01.999 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:36:01.999 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:02.000 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.997+0000 7f274b7fe640 1 -- 172.21.15.72:0/3004618437 <== mon.2 v1:172.21.15.72:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f275801aa70 con 0x7f275c006e00 2024-03-29T11:36:02.000 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.998+0000 7f274b7fe640 1 -- 172.21.15.72:0/3004618437 <== mon.2 v1:172.21.15.72:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f2758054780 con 0x7f275c006e00 2024-03-29T11:36:02.000 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.998+0000 7f274b7fe640 10 client.4262.objecter ms_dispatch 0x7f2764000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:02.000 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.998+0000 7f274b7fe640 3 client.4262.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:02.000 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.998+0000 7f274b7fe640 3 client.4262.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:02.000 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.998+0000 7f274b7fe640 20 client.4262.objecter dump_active .. 0 homeless 2024-03-29T11:36:02.000 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:01.999+0000 7f276259c640 10 monclient: get_auth_request con 0x7f275006e2b0 auth_method 0 2024-03-29T11:36:02.000 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:02.000+0000 7f274b7fe640 10 client.4262.objecter ms_handle_connect 0x7f275006e2b0 2024-03-29T11:36:02.007 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:02.008+0000 7f274b7fe640 1 -- 172.21.15.72:0/3004618437 <== mon.2 v1:172.21.15.72:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2758050a20 con 0x7f275c006e00 2024-03-29T11:36:02.007 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:02.008+0000 7f274b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:02.007 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:02.008+0000 7f274b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:02.337 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:02.338+0000 7f27497fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:02.337 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:02.339+0000 7f27497fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:02.337 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:02.339+0000 7f27497fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:36:02.338 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:02.339+0000 7f27497fa640 1 -- 172.21.15.72:0/3004618437 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f275c0332d0 con 0x7f275c006e00 2024-03-29T11:36:02.338 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:02.340+0000 7f274b7fe640 1 -- 172.21.15.72:0/3004618437 <== mon.2 v1:172.21.15.72:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f2758050bc0 con 0x7f275c006e00 2024-03-29T11:36:02.338 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:02.340+0000 7f274b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:02.339 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:02.340+0000 7f274b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:02.339 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:36:02.339 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:02.342 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:02.343+0000 7f27497fa640 10 librados: watch_flush enter 2024-03-29T11:36:02.342 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:02.344+0000 7f27497fa640 10 librados: watch_flush exit 2024-03-29T11:36:02.342 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:02.344+0000 7f27497fa640 20 client.4262.objecter shutdown clearing up homeless session... 2024-03-29T11:36:02.342 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:02.344+0000 7f27497fa640 10 client.4262.objecter successfully canceled tick 2024-03-29T11:36:02.344 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:02.345+0000 7f27497fa640 1 -- 172.21.15.72:0/3004618437 >> v1:172.21.15.72:6816/1578866102 conn(0x7f275006e2b0 legacy=0x7f2750070770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:02.344 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:02.346+0000 7f27497fa640 10 monclient: shutdown 2024-03-29T11:36:02.344 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:02.346+0000 7f27497fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:02.345 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:02.346+0000 7f27497fa640 1 -- 172.21.15.72:0/3004618437 >> v1:172.21.15.72:6790/0 conn(0x7f275c006e00 legacy=0x7f275c0328c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:02.348 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:02.349+0000 7f27497fa640 1 -- 172.21.15.72:0/3004618437 shutdown_connections 2024-03-29T11:36:02.348 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:02.349+0000 7f27497fa640 1 -- 172.21.15.72:0/3004618437 >> 172.21.15.72:0/3004618437 conn(0x7f275c005bd0 msgr2=0x7f275c0c8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:02.353 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:02.354+0000 7f27497fa640 1 -- 172.21.15.72:0/3004618437 shutdown_connections 2024-03-29T11:36:02.353 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:02.354+0000 7f27497fa640 1 -- 172.21.15.72:0/3004618437 wait complete. 2024-03-29T11:36:02.353 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:02.354+0000 7f27497fa640 1 librados: shutdown 2024-03-29T11:36:02.370 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:09.177 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:09.329 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.326+0000 7f134df09640 10 monclient: get_monmap_and_config 2024-03-29T11:36:09.329 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.326+0000 7f134df09640 10 monclient: build_initial_monmap 2024-03-29T11:36:09.329 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.326+0000 7f134df09640 10 monclient: monmap: 2024-03-29T11:36:09.329 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:36:09.329 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:36:09.329 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:36:09.327361+0000 2024-03-29T11:36:09.329 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:36:09.327361+0000 2024-03-29T11:36:09.329 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:09.329 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:09.330 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:36:09.330 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:36:09.330 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:36:09.330 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:09.330 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.327+0000 7f134df09640 1 Processor -- start 2024-03-29T11:36:09.330 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.327+0000 7f134df09640 1 -- start start 2024-03-29T11:36:09.330 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.327+0000 7f134df09640 10 monclient: init 2024-03-29T11:36:09.330 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.328+0000 7f134df09640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:09.330 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.328+0000 7f134df09640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:36:09.330 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.328+0000 7f134df09640 10 monclient(hunting): picked mon.noname-a con 0x7f1340034850 addr v1:172.21.15.72:6789/0 2024-03-29T11:36:09.330 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.328+0000 7f134df09640 10 monclient(hunting): picked mon.noname-b con 0x7f13400c9550 addr v1:172.21.15.72:6790/0 2024-03-29T11:36:09.330 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.328+0000 7f134df09640 10 monclient(hunting): picked mon.noname-c con 0x7f134002aac0 addr v1:172.21.15.89:6789/0 2024-03-29T11:36:09.330 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.328+0000 7f134df09640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f134002c440 con 0x7f1340034850 2024-03-29T11:36:09.330 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.328+0000 7f134df09640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f134002aed0 con 0x7f134002aac0 2024-03-29T11:36:09.330 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.328+0000 7f134df09640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f134002b190 con 0x7f13400c9550 2024-03-29T11:36:09.331 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.328+0000 7f134df09640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:09.331 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.328+0000 7f134df09640 10 monclient(hunting): authenticate will time out at 928.616978s 2024-03-29T11:36:09.331 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.329+0000 7f1347fff640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f13400c9550 0x7f13400cbea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:40014/0 (socket says 172.21.15.72:40014) 2024-03-29T11:36:09.331 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.329+0000 7f1347fff640 1 -- 172.21.15.72:0/2400537433 learned_addr learned my addr 172.21.15.72:0/2400537433 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:36:09.331 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.329+0000 7f13477fe640 1 -- 172.21.15.72:0/2400537433 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3470855624 0 0) 0x7f134002b190 con 0x7f13400c9550 2024-03-29T11:36:09.331 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.329+0000 7f13477fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:09.331 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.329+0000 7f13477fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:09.331 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.329+0000 7f13477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:09.331 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.329+0000 7f13477fe640 10 monclient(hunting): my global_id is 4265 2024-03-29T11:36:09.331 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.329+0000 7f13477fe640 1 -- 172.21.15.72:0/2400537433 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1338003520 con 0x7f13400c9550 2024-03-29T11:36:09.331 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.329+0000 7f13477fe640 1 -- 172.21.15.72:0/2400537433 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1733189162 0 0) 0x7f1338003520 con 0x7f13400c9550 2024-03-29T11:36:09.331 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.329+0000 7f13477fe640 1 -- 172.21.15.72:0/2400537433 >> v1:172.21.15.89:6789/0 conn(0x7f134002aac0 legacy=0x7f13400cc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:09.331 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.330+0000 7f13477fe640 1 -- 172.21.15.72:0/2400537433 >> v1:172.21.15.72:6789/0 conn(0x7f1340034850 legacy=0x7f13400c8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:09.331 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.330+0000 7f13477fe640 10 monclient: _finish_hunting 0 2024-03-29T11:36:09.331 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.330+0000 7f13477fe640 1 monclient: found mon.noname-b 2024-03-29T11:36:09.332 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.330+0000 7f13477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:09.332 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.330+0000 7f13477fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.72:6790/0 2024-03-29T11:36:09.332 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.330+0000 7f13477fe640 1 -- 172.21.15.72:0/2400537433 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1340033060 con 0x7f13400c9550 2024-03-29T11:36:09.332 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.330+0000 7f13477fe640 10 monclient: _finish_auth 0 2024-03-29T11:36:09.332 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.330+0000 7f13477fe640 10 monclient: _check_auth_tickets 2024-03-29T11:36:09.332 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.330+0000 7f13477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:09.332 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.330+0000 7f13477fe640 1 -- 172.21.15.72:0/2400537433 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f1334001690 con 0x7f13400c9550 2024-03-29T11:36:09.332 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.330+0000 7f13477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:09.332 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.330+0000 7f13477fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:09.332 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.330+0000 7f13477fe640 10 monclient: dump: 2024-03-29T11:36:09.332 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:36:09.332 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:36:09.332 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:09.332 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:09.332 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:09.332 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:09.333 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:36:09.333 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:36:09.333 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:36:09.333 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:09.333 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.330+0000 7f134df09640 5 monclient: authenticate success, global_id 4265 2024-03-29T11:36:09.333 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.330+0000 7f134df09640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:09.339 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.330+0000 7f13477fe640 1 -- 172.21.15.72:0/2400537433 <== mon.2 v1:172.21.15.72:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1334001860 con 0x7f13400c9550 2024-03-29T11:36:09.339 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.330+0000 7f13477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:09.339 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.330+0000 7f134df09640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:09.339 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.330+0000 7f134df09640 4 set_mon_vals no callback set 2024-03-29T11:36:09.339 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.330+0000 7f13477fe640 1 -- 172.21.15.72:0/2400537433 <== mon.2 v1:172.21.15.72:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f1334001cb0 con 0x7f13400c9550 2024-03-29T11:36:09.339 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.330+0000 7f13477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:09.339 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.330+0000 7f13477fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:09.339 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.330+0000 7f13477fe640 10 monclient: dump: 2024-03-29T11:36:09.340 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:36:09.340 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:36:09.340 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:09.340 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:09.340 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:09.340 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:09.340 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:36:09.340 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:36:09.340 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:36:09.340 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:09.340 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.331+0000 7f134df09640 10 monclient: shutdown 2024-03-29T11:36:09.340 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.331+0000 7f134df09640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:09.340 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.331+0000 7f134df09640 1 -- 172.21.15.72:0/2400537433 >> v1:172.21.15.72:6790/0 conn(0x7f13400c9550 legacy=0x7f13400cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:09.340 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.331+0000 7f134df09640 1 -- 172.21.15.72:0/2400537433 shutdown_connections 2024-03-29T11:36:09.340 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.331+0000 7f134df09640 1 -- 172.21.15.72:0/2400537433 >> 172.21.15.72:0/2400537433 conn(0x7f13400058f0 msgr2=0x7f1340005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:09.341 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.332+0000 7f134df09640 1 -- 172.21.15.72:0/2400537433 shutdown_connections 2024-03-29T11:36:09.341 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.332+0000 7f134df09640 1 -- 172.21.15.72:0/2400537433 wait complete. 2024-03-29T11:36:09.341 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.333+0000 7f134df09640 10 monclient: build_initial_monmap 2024-03-29T11:36:09.341 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.333+0000 7f134df09640 10 monclient: monmap: 2024-03-29T11:36:09.341 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:36:09.341 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:09.341 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:36:09.341 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:36:09.341 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:09.341 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:09.341 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:36:09.341 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:36:09.341 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:36:09.341 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:09.341 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.333+0000 7f134df09640 1 librados: starting msgr at 2024-03-29T11:36:09.342 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.333+0000 7f134df09640 1 librados: starting objecter 2024-03-29T11:36:09.342 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.333+0000 7f134df09640 1 Processor -- start 2024-03-29T11:36:09.342 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.333+0000 7f134df09640 1 -- start start 2024-03-29T11:36:09.342 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.333+0000 7f134df09640 1 librados: setting wanted keys 2024-03-29T11:36:09.342 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.333+0000 7f134df09640 1 librados: calling monclient init 2024-03-29T11:36:09.342 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.333+0000 7f134df09640 10 monclient: init 2024-03-29T11:36:09.342 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.334+0000 7f134df09640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:09.342 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.334+0000 7f134df09640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:36:09.342 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.334+0000 7f134df09640 10 monclient(hunting): picked mon.noname-c con 0x7f134002aac0 addr v1:172.21.15.72:6790/0 2024-03-29T11:36:09.342 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.334+0000 7f134cf07640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f134002aac0 0x7f13400cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:40524/0 (socket says 172.21.15.72:40524) 2024-03-29T11:36:09.342 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.334+0000 7f134cf07640 1 -- 172.21.15.72:0/4229621637 learned_addr learned my addr 172.21.15.72:0/4229621637 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:36:09.342 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.334+0000 7f134df09640 10 monclient(hunting): picked mon.noname-a con 0x7f1340034850 addr v1:172.21.15.72:6789/0 2024-03-29T11:36:09.342 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.334+0000 7f134df09640 10 monclient(hunting): picked mon.noname-b con 0x7f13400c9550 addr v1:172.21.15.89:6789/0 2024-03-29T11:36:09.342 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.335+0000 7f134df09640 1 -- 172.21.15.72:0/4229621637 --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13400cfb10 con 0x7f1340034850 2024-03-29T11:36:09.343 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.335+0000 7f134df09640 1 -- 172.21.15.72:0/4229621637 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13400ce020 con 0x7f13400c9550 2024-03-29T11:36:09.343 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.335+0000 7f134df09640 1 -- 172.21.15.72:0/4229621637 --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13400ce1b0 con 0x7f134002aac0 2024-03-29T11:36:09.343 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.335+0000 7f134df09640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:09.343 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.335+0000 7f134df09640 10 monclient(hunting): authenticate will time out at 928.623149s 2024-03-29T11:36:09.343 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.335+0000 7f13457fa640 10 client.?.objecter ms_handle_connect 0x7f134002aac0 2024-03-29T11:36:09.343 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.335+0000 7f13457fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:09.343 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.335+0000 7f13457fa640 10 client.?.objecter ms_handle_connect 0x7f13400c9550 2024-03-29T11:36:09.343 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.335+0000 7f13457fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:09.343 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.335+0000 7f13457fa640 10 client.?.objecter ms_handle_connect 0x7f1340034850 2024-03-29T11:36:09.343 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.335+0000 7f13457fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:09.343 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.336+0000 7f13457fa640 1 -- 172.21.15.72:0/4229621637 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3490654473 0 0) 0x7f13400ce1b0 con 0x7f134002aac0 2024-03-29T11:36:09.343 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.336+0000 7f13457fa640 10 monclient(hunting): _negotiate 2024-03-29T11:36:09.343 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.336+0000 7f13457fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:09.344 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.336+0000 7f13457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:09.344 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.336+0000 7f13457fa640 10 monclient(hunting): my global_id is 4268 2024-03-29T11:36:09.344 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.336+0000 7f13457fa640 1 -- 172.21.15.72:0/4229621637 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f132c004eb0 con 0x7f134002aac0 2024-03-29T11:36:09.344 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.336+0000 7f13457fa640 1 -- 172.21.15.72:0/4229621637 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3153201179 0 0) 0x7f13400cfb10 con 0x7f1340034850 2024-03-29T11:36:09.344 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.336+0000 7f13457fa640 10 monclient(hunting): _negotiate 2024-03-29T11:36:09.344 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.336+0000 7f13457fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:09.344 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.336+0000 7f13457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:09.344 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.336+0000 7f13457fa640 10 monclient(hunting): my global_id is 4269 2024-03-29T11:36:09.344 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.336+0000 7f13457fa640 1 -- 172.21.15.72:0/4229621637 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13400ce1b0 con 0x7f1340034850 2024-03-29T11:36:09.344 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.336+0000 7f13457fa640 1 -- 172.21.15.72:0/4229621637 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 822002458 0 0) 0x7f13400ce020 con 0x7f13400c9550 2024-03-29T11:36:09.344 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.336+0000 7f13457fa640 10 monclient(hunting): _negotiate 2024-03-29T11:36:09.344 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.336+0000 7f13457fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:09.344 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.336+0000 7f13457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:09.344 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.336+0000 7f13457fa640 10 monclient(hunting): my global_id is 4273 2024-03-29T11:36:09.344 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.336+0000 7f13457fa640 1 -- 172.21.15.72:0/4229621637 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13400cfb10 con 0x7f13400c9550 2024-03-29T11:36:09.344 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.336+0000 7f13457fa640 1 -- 172.21.15.72:0/4229621637 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3390074176 0 0) 0x7f132c004eb0 con 0x7f134002aac0 2024-03-29T11:36:09.345 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.336+0000 7f13457fa640 1 -- 172.21.15.72:0/4229621637 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13400ce020 con 0x7f134002aac0 2024-03-29T11:36:09.345 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.336+0000 7f13457fa640 1 -- 172.21.15.72:0/4229621637 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f133c001a30 con 0x7f134002aac0 2024-03-29T11:36:09.345 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.336+0000 7f13457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:09.345 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.336+0000 7f13457fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:09.345 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.336+0000 7f13457fa640 10 monclient(hunting): dump: 2024-03-29T11:36:09.345 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:36:09.345 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:36:09.345 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:09.345 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:09.345 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:09.345 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:09.345 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:36:09.345 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:36:09.345 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:36:09.345 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:09.346 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.336+0000 7f13457fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:09.346 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.337+0000 7f13457fa640 1 -- 172.21.15.72:0/4229621637 <== mon.2 v1:172.21.15.72:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3124773737 0 0) 0x7f13400ce020 con 0x7f134002aac0 2024-03-29T11:36:09.346 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.337+0000 7f13457fa640 1 -- 172.21.15.72:0/4229621637 >> v1:172.21.15.89:6789/0 conn(0x7f13400c9550 legacy=0x7f1340162ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:09.346 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.337+0000 7f13457fa640 1 -- 172.21.15.72:0/4229621637 >> v1:172.21.15.72:6789/0 conn(0x7f1340034850 legacy=0x7f134015b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:09.346 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.337+0000 7f13457fa640 10 monclient: _finish_hunting 0 2024-03-29T11:36:09.346 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.337+0000 7f13457fa640 1 monclient: found mon.c 2024-03-29T11:36:09.346 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.337+0000 7f13457fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:09.346 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.337+0000 7f13457fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:36:09.346 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.337+0000 7f13457fa640 1 -- 172.21.15.72:0/4229621637 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1340038150 con 0x7f134002aac0 2024-03-29T11:36:09.346 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.337+0000 7f13457fa640 10 monclient: _finish_auth 0 2024-03-29T11:36:09.346 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.337+0000 7f13457fa640 10 monclient: _check_auth_tickets 2024-03-29T11:36:09.346 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.337+0000 7f13457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:09.346 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.337+0000 7f134df09640 5 monclient: authenticate success, global_id 4268 2024-03-29T11:36:09.346 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.337+0000 7f134df09640 10 monclient: _renew_subs 2024-03-29T11:36:09.346 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.337+0000 7f134df09640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:36:09.347 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.337+0000 7f134df09640 1 -- 172.21.15.72:0/4229621637 --> v1:172.21.15.72:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1340038400 con 0x7f134002aac0 2024-03-29T11:36:09.347 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.337+0000 7f134df09640 10 client.4268.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:09.347 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.337+0000 7f134df09640 10 monclient: _renew_subs 2024-03-29T11:36:09.347 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.337+0000 7f134df09640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:36:09.347 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.337+0000 7f134df09640 1 -- 172.21.15.72:0/4229621637 --> v1:172.21.15.72:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f134015c270 con 0x7f134002aac0 2024-03-29T11:36:09.347 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.337+0000 7f134df09640 1 librados: init done 2024-03-29T11:36:09.347 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.338+0000 7f134df09640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:09.347 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.338+0000 7f134df09640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:09.347 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.338+0000 7f134df09640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:36:09.347 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.338+0000 7f13457fa640 1 -- 172.21.15.72:0/4229621637 <== mon.2 v1:172.21.15.72:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f133c001c70 con 0x7f134002aac0 2024-03-29T11:36:09.347 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.338+0000 7f134df09640 1 -- 172.21.15.72:0/4229621637 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1340004cd0 con 0x7f134002aac0 2024-03-29T11:36:09.347 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.339+0000 7f13457fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:09.347 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.339+0000 7f13457fa640 1 -- 172.21.15.72:0/4229621637 <== mon.2 v1:172.21.15.72:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f133c003a60 con 0x7f134002aac0 2024-03-29T11:36:09.347 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.339+0000 7f13457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:09.347 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.339+0000 7f13457fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:09.347 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.339+0000 7f13457fa640 10 monclient: dump: 2024-03-29T11:36:09.348 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:36:09.348 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:36:09.348 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:09.348 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:09.348 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:09.348 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:09.348 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:36:09.348 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:36:09.348 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:36:09.348 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:09.348 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.339+0000 7f1346ffd640 4 set_mon_vals no callback set 2024-03-29T11:36:09.348 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.340+0000 7f13457fa640 1 -- 172.21.15.72:0/4229621637 <== mon.2 v1:172.21.15.72:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f133c003c40 con 0x7f134002aac0 2024-03-29T11:36:09.350 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.341+0000 7f1347fff640 10 monclient: get_auth_request con 0x7f132c06de80 auth_method 0 2024-03-29T11:36:09.350 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.341+0000 7f13457fa640 10 client.4268.objecter ms_handle_connect 0x7f132c06de80 2024-03-29T11:36:09.350 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.341+0000 7f13457fa640 1 -- 172.21.15.72:0/4229621637 <== mon.2 v1:172.21.15.72:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f133c084ad0 con 0x7f134002aac0 2024-03-29T11:36:09.350 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.341+0000 7f13457fa640 10 client.4268.objecter ms_dispatch 0x7f1348000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:09.350 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.341+0000 7f13457fa640 3 client.4268.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:09.350 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.341+0000 7f13457fa640 3 client.4268.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:09.350 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.341+0000 7f13457fa640 20 client.4268.objecter dump_active .. 0 homeless 2024-03-29T11:36:09.350 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.347+0000 7f13457fa640 1 -- 172.21.15.72:0/4229621637 <== mon.2 v1:172.21.15.72:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f133c0503b0 con 0x7f134002aac0 2024-03-29T11:36:09.350 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.347+0000 7f13457fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:09.350 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.347+0000 7f13457fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:09.714 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.715+0000 7f134df09640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:09.714 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.715+0000 7f134df09640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:09.715 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.715+0000 7f134df09640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:36:09.715 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.715+0000 7f134df09640 1 -- 172.21.15.72:0/4229621637 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f13400086f0 con 0x7f134002aac0 2024-03-29T11:36:09.715 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.716+0000 7f13457fa640 1 -- 172.21.15.72:0/4229621637 <== mon.2 v1:172.21.15.72:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f133c0522e0 con 0x7f134002aac0 2024-03-29T11:36:09.715 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.716+0000 7f13457fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:09.715 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.716+0000 7f13457fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:09.716 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:36:09.716 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:09.719 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.720+0000 7f134df09640 10 librados: watch_flush enter 2024-03-29T11:36:09.719 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.720+0000 7f134df09640 10 librados: watch_flush exit 2024-03-29T11:36:09.719 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.720+0000 7f134df09640 20 client.4268.objecter shutdown clearing up homeless session... 2024-03-29T11:36:09.719 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.720+0000 7f134df09640 10 client.4268.objecter successfully canceled tick 2024-03-29T11:36:09.719 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.720+0000 7f134df09640 1 -- 172.21.15.72:0/4229621637 >> v1:172.21.15.72:6816/1578866102 conn(0x7f132c06de80 legacy=0x7f132c070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:09.719 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.720+0000 7f134df09640 10 monclient: shutdown 2024-03-29T11:36:09.719 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.720+0000 7f134df09640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:09.720 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.720+0000 7f134df09640 1 -- 172.21.15.72:0/4229621637 >> v1:172.21.15.72:6790/0 conn(0x7f134002aac0 legacy=0x7f13400cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:09.720 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.720+0000 7f134df09640 1 -- 172.21.15.72:0/4229621637 shutdown_connections 2024-03-29T11:36:09.720 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.720+0000 7f134df09640 1 -- 172.21.15.72:0/4229621637 >> 172.21.15.72:0/4229621637 conn(0x7f13400058f0 msgr2=0x7f13400043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:09.720 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.721+0000 7f134df09640 1 -- 172.21.15.72:0/4229621637 shutdown_connections 2024-03-29T11:36:09.720 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.721+0000 7f134df09640 1 -- 172.21.15.72:0/4229621637 wait complete. 2024-03-29T11:36:09.720 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:09.721+0000 7f134df09640 1 librados: shutdown 2024-03-29T11:36:09.734 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:16.541 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:16.704 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.705+0000 7f4036525640 10 monclient: get_monmap_and_config 2024-03-29T11:36:16.704 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.705+0000 7f4036525640 10 monclient: build_initial_monmap 2024-03-29T11:36:16.704 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.705+0000 7f4036525640 10 monclient: monmap: 2024-03-29T11:36:16.704 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:36:16.704 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:36:16.705 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:36:16.706629+0000 2024-03-29T11:36:16.705 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:36:16.706629+0000 2024-03-29T11:36:16.705 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:16.705 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:16.705 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:36:16.705 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:36:16.705 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:36:16.705 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:16.706 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.706+0000 7f4036525640 1 Processor -- start 2024-03-29T11:36:16.706 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.706+0000 7f4036525640 1 -- start start 2024-03-29T11:36:16.706 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.706+0000 7f4036525640 10 monclient: init 2024-03-29T11:36:16.710 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.707+0000 7f4036525640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:16.710 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.707+0000 7f4036525640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:36:16.710 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.707+0000 7f4036525640 10 monclient(hunting): picked mon.noname-b con 0x7f40301457f0 addr v1:172.21.15.72:6790/0 2024-03-29T11:36:16.710 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.707+0000 7f4036525640 10 monclient(hunting): picked mon.noname-a con 0x7f403013f650 addr v1:172.21.15.72:6789/0 2024-03-29T11:36:16.710 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.708+0000 7f4036525640 10 monclient(hunting): picked mon.noname-c con 0x7f40300a9d70 addr v1:172.21.15.89:6789/0 2024-03-29T11:36:16.710 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.708+0000 7f4036525640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40300aaef0 con 0x7f403013f650 2024-03-29T11:36:16.710 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.708+0000 7f4036525640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4030145bd0 con 0x7f40300a9d70 2024-03-29T11:36:16.710 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.708+0000 7f4036525640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4030145eb0 con 0x7f40301457f0 2024-03-29T11:36:16.711 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.708+0000 7f4036525640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:16.711 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.708+0000 7f4036525640 10 monclient(hunting): authenticate will time out at 935.996157s 2024-03-29T11:36:16.711 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.708+0000 7f4034ac2640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f40300a9d70 0x7f403014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:51710/0 (socket says 172.21.15.72:51710) 2024-03-29T11:36:16.711 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.708+0000 7f4034ac2640 1 -- 172.21.15.72:0/42155657 learned_addr learned my addr 172.21.15.72:0/42155657 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:36:16.711 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.708+0000 7f402effd640 1 -- 172.21.15.72:0/42155657 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 362166431 0 0) 0x7f4030145eb0 con 0x7f40301457f0 2024-03-29T11:36:16.711 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.708+0000 7f402effd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:16.711 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.708+0000 7f402effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:16.711 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.708+0000 7f402effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:16.711 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.708+0000 7f402effd640 10 monclient(hunting): my global_id is 4271 2024-03-29T11:36:16.711 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.708+0000 7f402effd640 1 -- 172.21.15.72:0/42155657 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4018003520 con 0x7f40301457f0 2024-03-29T11:36:16.711 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.708+0000 7f402effd640 1 -- 172.21.15.72:0/42155657 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4123801131 0 0) 0x7f4030145bd0 con 0x7f40300a9d70 2024-03-29T11:36:16.711 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.708+0000 7f402effd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:16.711 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.708+0000 7f402effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:16.711 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.708+0000 7f402effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:16.711 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.708+0000 7f402effd640 10 monclient(hunting): my global_id is 4276 2024-03-29T11:36:16.712 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.708+0000 7f402effd640 1 -- 172.21.15.72:0/42155657 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4030145eb0 con 0x7f40300a9d70 2024-03-29T11:36:16.712 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.709+0000 7f402effd640 1 -- 172.21.15.72:0/42155657 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 682918420 0 0) 0x7f4018003520 con 0x7f40301457f0 2024-03-29T11:36:16.712 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.709+0000 7f402effd640 1 -- 172.21.15.72:0/42155657 >> v1:172.21.15.89:6789/0 conn(0x7f40300a9d70 legacy=0x7f403014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.712 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.709+0000 7f402effd640 1 -- 172.21.15.72:0/42155657 >> v1:172.21.15.72:6789/0 conn(0x7f403013f650 legacy=0x7f40300b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.712 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.709+0000 7f402effd640 10 monclient: _finish_hunting 0 2024-03-29T11:36:16.712 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.709+0000 7f402effd640 1 monclient: found mon.noname-b 2024-03-29T11:36:16.712 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.709+0000 7f402effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:16.712 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.709+0000 7f402effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.72:6790/0 2024-03-29T11:36:16.712 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.709+0000 7f402effd640 1 -- 172.21.15.72:0/42155657 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4030146340 con 0x7f40301457f0 2024-03-29T11:36:16.712 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.709+0000 7f402effd640 10 monclient: _finish_auth 0 2024-03-29T11:36:16.712 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.709+0000 7f402effd640 10 monclient: _check_auth_tickets 2024-03-29T11:36:16.712 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.709+0000 7f402effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:16.712 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.709+0000 7f402effd640 1 -- 172.21.15.72:0/42155657 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f4020001690 con 0x7f40301457f0 2024-03-29T11:36:16.712 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.709+0000 7f402effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:16.713 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.709+0000 7f402effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:16.713 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.709+0000 7f402effd640 10 monclient: dump: 2024-03-29T11:36:16.713 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:36:16.713 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:36:16.713 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:16.713 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:16.713 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:16.713 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:16.713 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:36:16.713 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:36:16.713 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:36:16.713 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:16.713 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.709+0000 7f4036525640 5 monclient: authenticate success, global_id 4271 2024-03-29T11:36:16.713 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.709+0000 7f4036525640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:16.713 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.710+0000 7f402effd640 1 -- 172.21.15.72:0/42155657 <== mon.2 v1:172.21.15.72:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40200017f0 con 0x7f40301457f0 2024-03-29T11:36:16.714 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.710+0000 7f402effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:16.714 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.710+0000 7f402effd640 1 -- 172.21.15.72:0/42155657 <== mon.2 v1:172.21.15.72:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f4020001c40 con 0x7f40301457f0 2024-03-29T11:36:16.714 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.710+0000 7f402effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:16.714 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.710+0000 7f402effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:16.714 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.710+0000 7f402effd640 10 monclient: dump: 2024-03-29T11:36:16.714 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:36:16.714 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:36:16.714 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:16.714 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:16.714 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:16.714 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:16.714 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:36:16.714 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:36:16.714 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:36:16.714 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:16.715 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.710+0000 7f4036525640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:16.715 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.710+0000 7f4036525640 4 set_mon_vals no callback set 2024-03-29T11:36:16.715 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.711+0000 7f4036525640 10 monclient: shutdown 2024-03-29T11:36:16.715 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.711+0000 7f4036525640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:16.715 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.711+0000 7f4036525640 1 -- 172.21.15.72:0/42155657 >> v1:172.21.15.72:6790/0 conn(0x7f40301457f0 legacy=0x7f403013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.715 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.711+0000 7f4036525640 1 -- 172.21.15.72:0/42155657 shutdown_connections 2024-03-29T11:36:16.715 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.711+0000 7f4036525640 1 -- 172.21.15.72:0/42155657 >> 172.21.15.72:0/42155657 conn(0x7f4030066a00 msgr2=0x7f4030066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:16.715 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.711+0000 7f4036525640 1 -- 172.21.15.72:0/42155657 shutdown_connections 2024-03-29T11:36:16.715 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.711+0000 7f4036525640 1 -- 172.21.15.72:0/42155657 wait complete. 2024-03-29T11:36:16.721 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.712+0000 7f4036525640 10 monclient: build_initial_monmap 2024-03-29T11:36:16.722 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.712+0000 7f4036525640 10 monclient: monmap: 2024-03-29T11:36:16.722 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:36:16.722 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:16.722 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:36:16.722 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:36:16.722 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:16.722 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:16.722 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:36:16.722 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:36:16.722 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:36:16.722 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:16.722 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.713+0000 7f4036525640 1 librados: starting msgr at 2024-03-29T11:36:16.722 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.713+0000 7f4036525640 1 librados: starting objecter 2024-03-29T11:36:16.723 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.713+0000 7f4036525640 1 Processor -- start 2024-03-29T11:36:16.723 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.713+0000 7f4036525640 1 -- start start 2024-03-29T11:36:16.723 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.713+0000 7f4036525640 1 librados: setting wanted keys 2024-03-29T11:36:16.723 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.713+0000 7f4036525640 1 librados: calling monclient init 2024-03-29T11:36:16.723 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.713+0000 7f4036525640 10 monclient: init 2024-03-29T11:36:16.723 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.714+0000 7f4036525640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:16.723 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.714+0000 7f4036525640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:36:16.723 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.714+0000 7f4036525640 10 monclient(hunting): picked mon.noname-c con 0x7f40300a9d70 addr v1:172.21.15.72:6790/0 2024-03-29T11:36:16.723 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.714+0000 7f4036525640 10 monclient(hunting): picked mon.noname-b con 0x7f403013f650 addr v1:172.21.15.89:6789/0 2024-03-29T11:36:16.723 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.714+0000 7f4036525640 10 monclient(hunting): picked mon.noname-a con 0x7f40301457f0 addr v1:172.21.15.72:6789/0 2024-03-29T11:36:16.723 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.714+0000 7f4036525640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4030142560 con 0x7f40301457f0 2024-03-29T11:36:16.723 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.714+0000 7f4036525640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f403013b510 con 0x7f403013f650 2024-03-29T11:36:16.723 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.714+0000 7f4036525640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4030071f90 con 0x7f40300a9d70 2024-03-29T11:36:16.723 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.714+0000 7f4036525640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:16.723 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.714+0000 7f4036525640 10 monclient(hunting): authenticate will time out at 936.002803s 2024-03-29T11:36:16.724 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.714+0000 7f402ffff640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f40300a9d70 0x7f4030141e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:43074/0 (socket says 172.21.15.72:43074) 2024-03-29T11:36:16.724 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.714+0000 7f402ffff640 1 -- 172.21.15.72:0/550177527 learned_addr learned my addr 172.21.15.72:0/550177527 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:36:16.724 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.715+0000 7f402cff9640 10 client.?.objecter ms_handle_connect 0x7f403013f650 2024-03-29T11:36:16.724 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.715+0000 7f402cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:16.724 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.715+0000 7f402cff9640 10 client.?.objecter ms_handle_connect 0x7f40300a9d70 2024-03-29T11:36:16.724 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.715+0000 7f402cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:16.724 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.715+0000 7f402cff9640 1 -- 172.21.15.72:0/550177527 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2950798864 0 0) 0x7f403013b510 con 0x7f403013f650 2024-03-29T11:36:16.724 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.715+0000 7f402cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:36:16.724 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.715+0000 7f402cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:16.724 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.715+0000 7f402cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:16.724 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.715+0000 7f402cff9640 10 monclient(hunting): my global_id is 4279 2024-03-29T11:36:16.724 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.715+0000 7f402cff9640 1 -- 172.21.15.72:0/550177527 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f401c004eb0 con 0x7f403013f650 2024-03-29T11:36:16.724 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.715+0000 7f402cff9640 10 client.?.objecter ms_handle_connect 0x7f40301457f0 2024-03-29T11:36:16.724 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.716+0000 7f402cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:16.724 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.716+0000 7f402cff9640 1 -- 172.21.15.72:0/550177527 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1562100891 0 0) 0x7f401c004eb0 con 0x7f403013f650 2024-03-29T11:36:16.725 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.716+0000 7f402cff9640 1 -- 172.21.15.72:0/550177527 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f403013b510 con 0x7f403013f650 2024-03-29T11:36:16.725 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.716+0000 7f402cff9640 1 -- 172.21.15.72:0/550177527 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f4024001930 con 0x7f403013f650 2024-03-29T11:36:16.725 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.716+0000 7f402cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:16.725 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.716+0000 7f402cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:16.725 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.716+0000 7f402cff9640 10 monclient(hunting): dump: 2024-03-29T11:36:16.725 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:36:16.725 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:36:16.725 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:16.725 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:16.725 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:16.725 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:16.725 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:36:16.725 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:36:16.725 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:36:16.725 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:16.726 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.716+0000 7f402cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:16.726 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.716+0000 7f402cff9640 1 -- 172.21.15.72:0/550177527 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1902156041 0 0) 0x7f4030142560 con 0x7f40301457f0 2024-03-29T11:36:16.726 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.716+0000 7f402cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:36:16.726 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.716+0000 7f402cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:16.726 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.716+0000 7f402cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:16.726 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.716+0000 7f402cff9640 10 monclient(hunting): my global_id is 4275 2024-03-29T11:36:16.726 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.716+0000 7f402cff9640 1 -- 172.21.15.72:0/550177527 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f401c004eb0 con 0x7f40301457f0 2024-03-29T11:36:16.726 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.716+0000 7f402cff9640 1 -- 172.21.15.72:0/550177527 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2425589713 0 0) 0x7f4030071f90 con 0x7f40300a9d70 2024-03-29T11:36:16.726 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.716+0000 7f402cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:36:16.726 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.716+0000 7f402cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:16.726 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.716+0000 7f402cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:16.726 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.716+0000 7f402cff9640 10 monclient(hunting): my global_id is 4274 2024-03-29T11:36:16.726 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.716+0000 7f402cff9640 1 -- 172.21.15.72:0/550177527 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4030142560 con 0x7f40300a9d70 2024-03-29T11:36:16.726 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.717+0000 7f402cff9640 1 -- 172.21.15.72:0/550177527 <== mon.1 v1:172.21.15.89:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4004528229 0 0) 0x7f403013b510 con 0x7f403013f650 2024-03-29T11:36:16.726 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.717+0000 7f402cff9640 1 -- 172.21.15.72:0/550177527 >> v1:172.21.15.72:6790/0 conn(0x7f40300a9d70 legacy=0x7f4030141e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.727 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.717+0000 7f402cff9640 1 -- 172.21.15.72:0/550177527 >> v1:172.21.15.72:6789/0 conn(0x7f40301457f0 legacy=0x7f4030137810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:16.727 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.717+0000 7f402cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:36:16.727 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.717+0000 7f402cff9640 1 monclient: found mon.b 2024-03-29T11:36:16.727 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.717+0000 7f402cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:16.727 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.717+0000 7f402cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:36:16.727 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.717+0000 7f402cff9640 1 -- 172.21.15.72:0/550177527 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f403013b0f0 con 0x7f403013f650 2024-03-29T11:36:16.727 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.717+0000 7f402cff9640 10 monclient: _finish_auth 0 2024-03-29T11:36:16.727 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.717+0000 7f402cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:36:16.727 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.717+0000 7f402cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:16.727 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.717+0000 7f4036525640 5 monclient: authenticate success, global_id 4279 2024-03-29T11:36:16.727 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.717+0000 7f4036525640 10 monclient: _renew_subs 2024-03-29T11:36:16.727 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.717+0000 7f4036525640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:36:16.727 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.717+0000 7f4036525640 1 -- 172.21.15.72:0/550177527 --> v1:172.21.15.89:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f403006de90 con 0x7f403013f650 2024-03-29T11:36:16.727 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.717+0000 7f4036525640 10 client.4279.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:16.727 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.717+0000 7f4036525640 10 monclient: _renew_subs 2024-03-29T11:36:16.728 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.717+0000 7f4036525640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:36:16.728 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.717+0000 7f4036525640 1 -- 172.21.15.72:0/550177527 --> v1:172.21.15.89:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f40301db9a0 con 0x7f403013f650 2024-03-29T11:36:16.728 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.717+0000 7f4036525640 1 librados: init done 2024-03-29T11:36:16.728 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.718+0000 7f402cff9640 1 -- 172.21.15.72:0/550177527 <== mon.1 v1:172.21.15.89:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4024003610 con 0x7f403013f650 2024-03-29T11:36:16.728 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.718+0000 7f402cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:16.728 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.718+0000 7f402cff9640 1 -- 172.21.15.72:0/550177527 <== mon.1 v1:172.21.15.89:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f40240039e0 con 0x7f403013f650 2024-03-29T11:36:16.728 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.718+0000 7f402cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:16.728 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.718+0000 7f402cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:16.728 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.718+0000 7f402cff9640 10 monclient: dump: 2024-03-29T11:36:16.728 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:36:16.728 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:36:16.728 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:16.728 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:16.728 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:16.728 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:16.728 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:36:16.729 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:36:16.729 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:36:16.729 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:16.729 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.718+0000 7f402e7fc640 4 set_mon_vals no callback set 2024-03-29T11:36:16.729 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.719+0000 7f4036525640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:16.729 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.719+0000 7f4036525640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:16.729 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.719+0000 7f4036525640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:36:16.729 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.719+0000 7f4036525640 1 -- 172.21.15.72:0/550177527 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4010005080 con 0x7f403013f650 2024-03-29T11:36:16.729 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.720+0000 7f402cff9640 1 -- 172.21.15.72:0/550177527 <== mon.1 v1:172.21.15.89:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f4024003b40 con 0x7f403013f650 2024-03-29T11:36:16.729 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.721+0000 7f402cff9640 1 -- 172.21.15.72:0/550177527 <== mon.1 v1:172.21.15.89:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f4024083820 con 0x7f403013f650 2024-03-29T11:36:16.729 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.721+0000 7f402cff9640 10 client.4279.objecter ms_dispatch 0x7f4030000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:16.729 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.721+0000 7f402cff9640 3 client.4279.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:16.729 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.721+0000 7f402cff9640 3 client.4279.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:16.729 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.721+0000 7f402cff9640 20 client.4279.objecter dump_active .. 0 homeless 2024-03-29T11:36:16.729 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.722+0000 7f402ffff640 10 monclient: get_auth_request con 0x7f401c06de60 auth_method 0 2024-03-29T11:36:16.729 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.722+0000 7f402cff9640 10 client.4279.objecter ms_handle_connect 0x7f401c06de60 2024-03-29T11:36:16.730 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.728+0000 7f402cff9640 1 -- 172.21.15.72:0/550177527 <== mon.1 v1:172.21.15.89:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f402404f160 con 0x7f403013f650 2024-03-29T11:36:16.730 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.728+0000 7f402cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:16.730 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:16.728+0000 7f402cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:17.140 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:17.141+0000 7f4036525640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:17.140 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:17.141+0000 7f4036525640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:17.141 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:17.141+0000 7f4036525640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:36:17.141 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:17.141+0000 7f4036525640 1 -- 172.21.15.72:0/550177527 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4010005330 con 0x7f403013f650 2024-03-29T11:36:17.141 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:17.142+0000 7f402cff9640 1 -- 172.21.15.72:0/550177527 <== mon.1 v1:172.21.15.89:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f4024082ed0 con 0x7f403013f650 2024-03-29T11:36:17.141 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:17.143+0000 7f402cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:17.141 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:17.143+0000 7f402cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:17.142 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:36:17.142 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:17.146 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:17.146+0000 7f4036525640 10 librados: watch_flush enter 2024-03-29T11:36:17.146 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:17.146+0000 7f4036525640 10 librados: watch_flush exit 2024-03-29T11:36:17.146 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:17.146+0000 7f4036525640 20 client.4279.objecter shutdown clearing up homeless session... 2024-03-29T11:36:17.146 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:17.146+0000 7f4036525640 10 client.4279.objecter successfully canceled tick 2024-03-29T11:36:17.146 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:17.147+0000 7f4036525640 1 -- 172.21.15.72:0/550177527 >> v1:172.21.15.72:6816/1578866102 conn(0x7f401c06de60 legacy=0x7f401c070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:17.146 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:17.147+0000 7f4036525640 10 monclient: shutdown 2024-03-29T11:36:17.146 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:17.147+0000 7f4036525640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:17.146 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:17.147+0000 7f4036525640 1 -- 172.21.15.72:0/550177527 >> v1:172.21.15.89:6789/0 conn(0x7f403013f650 legacy=0x7f4030137100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:17.146 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:17.147+0000 7f4036525640 1 -- 172.21.15.72:0/550177527 shutdown_connections 2024-03-29T11:36:17.146 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:17.147+0000 7f4036525640 1 -- 172.21.15.72:0/550177527 >> 172.21.15.72:0/550177527 conn(0x7f4030066a00 msgr2=0x7f4030066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:17.146 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:17.147+0000 7f4036525640 1 -- 172.21.15.72:0/550177527 shutdown_connections 2024-03-29T11:36:17.147 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:17.147+0000 7f4036525640 1 -- 172.21.15.72:0/550177527 wait complete. 2024-03-29T11:36:17.147 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:17.147+0000 7f4036525640 1 librados: shutdown 2024-03-29T11:36:17.161 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:23.968 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:24.103 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.104+0000 7f25c624a640 10 monclient: get_monmap_and_config 2024-03-29T11:36:24.103 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.104+0000 7f25c624a640 10 monclient: build_initial_monmap 2024-03-29T11:36:24.103 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.104+0000 7f25c624a640 10 monclient: monmap: 2024-03-29T11:36:24.103 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:36:24.103 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:36:24.103 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:36:24.105252+0000 2024-03-29T11:36:24.103 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:36:24.105252+0000 2024-03-29T11:36:24.104 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:24.104 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:24.104 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:36:24.104 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:36:24.104 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:36:24.104 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:24.105 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.105+0000 7f25c624a640 1 Processor -- start 2024-03-29T11:36:24.105 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.105+0000 7f25c624a640 1 -- start start 2024-03-29T11:36:24.105 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.105+0000 7f25c624a640 10 monclient: init 2024-03-29T11:36:24.105 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.106+0000 7f25c624a640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:24.106 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.106+0000 7f25c624a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:36:24.106 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.106+0000 7f25c624a640 10 monclient(hunting): picked mon.noname-b con 0x7f25c00a9750 addr v1:172.21.15.72:6790/0 2024-03-29T11:36:24.106 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.106+0000 7f25c624a640 10 monclient(hunting): picked mon.noname-c con 0x7f25c013e1e0 addr v1:172.21.15.89:6789/0 2024-03-29T11:36:24.106 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.106+0000 7f25c624a640 10 monclient(hunting): picked mon.noname-a con 0x7f25c0071c30 addr v1:172.21.15.72:6789/0 2024-03-29T11:36:24.106 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.106+0000 7f25c624a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25c00ab300 con 0x7f25c0071c30 2024-03-29T11:36:24.106 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.106+0000 7f25c624a640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25c00ac2f0 con 0x7f25c013e1e0 2024-03-29T11:36:24.106 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.106+0000 7f25bf7fe640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f25c00a9750 0x7f25c013dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:41980/0 (socket says 172.21.15.72:41980) 2024-03-29T11:36:24.106 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.106+0000 7f25bf7fe640 1 -- 172.21.15.72:0/290348701 learned_addr learned my addr 172.21.15.72:0/290348701 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:36:24.106 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.106+0000 7f25c624a640 1 -- 172.21.15.72:0/290348701 --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25c00ad8d0 con 0x7f25c00a9750 2024-03-29T11:36:24.106 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.106+0000 7f25c624a640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:24.106 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.106+0000 7f25c624a640 10 monclient(hunting): authenticate will time out at 943.394722s 2024-03-29T11:36:24.106 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.106+0000 7f25be7fc640 1 -- 172.21.15.72:0/290348701 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1589470208 0 0) 0x7f25c00ad8d0 con 0x7f25c00a9750 2024-03-29T11:36:24.106 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.106+0000 7f25be7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:24.107 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.106+0000 7f25be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:24.107 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.106+0000 7f25be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:24.107 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.106+0000 7f25be7fc640 10 monclient(hunting): my global_id is 4277 2024-03-29T11:36:24.107 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.106+0000 7f25be7fc640 1 -- 172.21.15.72:0/290348701 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25a8003520 con 0x7f25c00a9750 2024-03-29T11:36:24.107 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.107+0000 7f25be7fc640 1 -- 172.21.15.72:0/290348701 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2071274999 0 0) 0x7f25a8003520 con 0x7f25c00a9750 2024-03-29T11:36:24.107 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.107+0000 7f25be7fc640 1 -- 172.21.15.72:0/290348701 >> v1:172.21.15.89:6789/0 conn(0x7f25c013e1e0 legacy=0x7f25c00ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:24.107 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.107+0000 7f25be7fc640 1 -- 172.21.15.72:0/290348701 >> v1:172.21.15.72:6789/0 conn(0x7f25c0071c30 legacy=0x7f25c00ace20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:36:24.108 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.109+0000 7f25be7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:36:24.108 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.109+0000 7f25be7fc640 1 monclient: found mon.noname-b 2024-03-29T11:36:24.108 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.109+0000 7f25be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:24.108 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.109+0000 7f25be7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.72:6790/0 2024-03-29T11:36:24.108 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.109+0000 7f25be7fc640 1 -- 172.21.15.72:0/290348701 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c00ada60 con 0x7f25c00a9750 2024-03-29T11:36:24.108 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.109+0000 7f25be7fc640 10 monclient: _finish_auth 0 2024-03-29T11:36:24.108 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.109+0000 7f25be7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:36:24.108 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.109+0000 7f25be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:24.108 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.109+0000 7f25be7fc640 1 -- 172.21.15.72:0/290348701 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f25b0001690 con 0x7f25c00a9750 2024-03-29T11:36:24.108 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.109+0000 7f25c624a640 5 monclient: authenticate success, global_id 4277 2024-03-29T11:36:24.109 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.109+0000 7f25be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:24.109 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.109+0000 7f25be7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:24.109 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.109+0000 7f25be7fc640 10 monclient: dump: 2024-03-29T11:36:24.109 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:36:24.109 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:36:24.109 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:24.109 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:24.109 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:24.109 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:24.109 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:36:24.109 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:36:24.109 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:36:24.109 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:24.109 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.109+0000 7f25c624a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:24.109 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.109+0000 7f25be7fc640 1 -- 172.21.15.72:0/290348701 <== mon.2 v1:172.21.15.72:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25b00018f0 con 0x7f25c00a9750 2024-03-29T11:36:24.110 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.109+0000 7f25be7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:24.110 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.109+0000 7f25c624a640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:24.110 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.109+0000 7f25c624a640 4 set_mon_vals no callback set 2024-03-29T11:36:24.110 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.109+0000 7f25be7fc640 1 -- 172.21.15.72:0/290348701 <== mon.2 v1:172.21.15.72:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f25b0001d40 con 0x7f25c00a9750 2024-03-29T11:36:24.110 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.109+0000 7f25be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:24.110 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.109+0000 7f25be7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:24.110 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.109+0000 7f25be7fc640 10 monclient: dump: 2024-03-29T11:36:24.110 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:36:24.110 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:36:24.110 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:24.110 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:24.110 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:24.110 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:24.110 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:36:24.110 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:36:24.111 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:36:24.111 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:24.119 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.110+0000 7f25c624a640 10 monclient: shutdown 2024-03-29T11:36:24.119 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.110+0000 7f25c624a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:24.119 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.110+0000 7f25c624a640 1 -- 172.21.15.72:0/290348701 >> v1:172.21.15.72:6790/0 conn(0x7f25c00a9750 legacy=0x7f25c013dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:24.119 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.110+0000 7f25c624a640 1 -- 172.21.15.72:0/290348701 shutdown_connections 2024-03-29T11:36:24.119 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.110+0000 7f25c624a640 1 -- 172.21.15.72:0/290348701 >> 172.21.15.72:0/290348701 conn(0x7f25c006b660 msgr2=0x7f25c006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:24.119 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.110+0000 7f25c624a640 1 -- 172.21.15.72:0/290348701 shutdown_connections 2024-03-29T11:36:24.120 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.110+0000 7f25c624a640 1 -- 172.21.15.72:0/290348701 wait complete. 2024-03-29T11:36:24.120 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.111+0000 7f25c624a640 10 monclient: build_initial_monmap 2024-03-29T11:36:24.120 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.111+0000 7f25c624a640 10 monclient: monmap: 2024-03-29T11:36:24.120 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:36:24.120 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:24.120 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:36:24.120 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:36:24.120 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:24.120 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:24.120 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:36:24.120 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:36:24.120 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:36:24.120 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:24.120 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.111+0000 7f25c624a640 1 librados: starting msgr at 2024-03-29T11:36:24.120 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.111+0000 7f25c624a640 1 librados: starting objecter 2024-03-29T11:36:24.121 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.111+0000 7f25c624a640 1 Processor -- start 2024-03-29T11:36:24.121 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.111+0000 7f25c624a640 1 -- start start 2024-03-29T11:36:24.121 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.111+0000 7f25c624a640 1 librados: setting wanted keys 2024-03-29T11:36:24.121 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.111+0000 7f25c624a640 1 librados: calling monclient init 2024-03-29T11:36:24.121 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.111+0000 7f25c624a640 10 monclient: init 2024-03-29T11:36:24.121 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.112+0000 7f25c624a640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:24.121 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.112+0000 7f25c624a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:36:24.121 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.113+0000 7f25c624a640 10 monclient(hunting): picked mon.noname-c con 0x7f25c0071c30 addr v1:172.21.15.72:6790/0 2024-03-29T11:36:24.121 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.113+0000 7f25c624a640 10 monclient(hunting): picked mon.noname-b con 0x7f25c00a9750 addr v1:172.21.15.89:6789/0 2024-03-29T11:36:24.121 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.113+0000 7f25beffd640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f25c00a9750 0x7f25c0133340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:41382/0 (socket says 172.21.15.72:41382) 2024-03-29T11:36:24.121 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.113+0000 7f25c624a640 10 monclient(hunting): picked mon.noname-a con 0x7f25c013e1e0 addr v1:172.21.15.72:6789/0 2024-03-29T11:36:24.121 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.113+0000 7f25c624a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25c01c9870 con 0x7f25c013e1e0 2024-03-29T11:36:24.121 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.113+0000 7f25c624a640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25c0134160 con 0x7f25c00a9750 2024-03-29T11:36:24.121 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.113+0000 7f25c624a640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25c01c9420 con 0x7f25c0071c30 2024-03-29T11:36:24.121 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.113+0000 7f25bf7fe640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f25c0071c30 0x7f25c00b1190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:42632/0 (socket says 172.21.15.72:42632) 2024-03-29T11:36:24.122 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.113+0000 7f25bf7fe640 1 -- 172.21.15.72:0/1123298873 learned_addr learned my addr 172.21.15.72:0/1123298873 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:36:24.122 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.113+0000 7f25c624a640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:24.122 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.113+0000 7f25c624a640 10 monclient(hunting): authenticate will time out at 943.401737s 2024-03-29T11:36:24.122 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.113+0000 7f25a7fff640 10 client.?.objecter ms_handle_connect 0x7f25c0071c30 2024-03-29T11:36:24.122 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.113+0000 7f25a7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:24.122 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.113+0000 7f25a7fff640 1 -- 172.21.15.72:0/1123298873 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1359154190 0 0) 0x7f25c01c9420 con 0x7f25c0071c30 2024-03-29T11:36:24.122 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.113+0000 7f25a7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:24.122 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.113+0000 7f25a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:24.122 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.113+0000 7f25a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:24.122 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.113+0000 7f25a7fff640 10 monclient(hunting): my global_id is 4280 2024-03-29T11:36:24.122 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.113+0000 7f25a7fff640 1 -- 172.21.15.72:0/1123298873 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25ac0092c0 con 0x7f25c0071c30 2024-03-29T11:36:24.122 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.114+0000 7f25a7fff640 1 -- 172.21.15.72:0/1123298873 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 132867852 0 0) 0x7f25ac0092c0 con 0x7f25c0071c30 2024-03-29T11:36:24.122 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.114+0000 7f25a7fff640 1 -- 172.21.15.72:0/1123298873 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25c01c9420 con 0x7f25c0071c30 2024-03-29T11:36:24.122 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.114+0000 7f25a7fff640 10 client.?.objecter ms_handle_connect 0x7f25c00a9750 2024-03-29T11:36:24.123 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.114+0000 7f25a7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:24.123 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.114+0000 7f25a7fff640 1 -- 172.21.15.72:0/1123298873 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f25b0001d40 con 0x7f25c0071c30 2024-03-29T11:36:24.123 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.114+0000 7f25a7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:24.123 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.114+0000 7f25a7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:24.123 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.114+0000 7f25a7fff640 10 monclient(hunting): dump: 2024-03-29T11:36:24.123 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:36:24.123 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:36:24.123 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:24.123 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:24.123 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:24.123 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:24.123 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:36:24.123 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:36:24.123 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:36:24.123 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:24.124 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.114+0000 7f25a7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:24.124 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.114+0000 7f25a7fff640 1 -- 172.21.15.72:0/1123298873 <== mon.2 v1:172.21.15.72:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2011549865 0 0) 0x7f25c01c9420 con 0x7f25c0071c30 2024-03-29T11:36:24.124 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.114+0000 7f25a7fff640 1 -- 172.21.15.72:0/1123298873 >> v1:172.21.15.89:6789/0 conn(0x7f25c00a9750 legacy=0x7f25c0133340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:24.124 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.114+0000 7f25a7fff640 1 -- 172.21.15.72:0/1123298873 >> v1:172.21.15.72:6789/0 conn(0x7f25c013e1e0 legacy=0x7f25c0133a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:36:24.124 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.115+0000 7f25a7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:36:24.124 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.115+0000 7f25a7fff640 1 monclient: found mon.c 2024-03-29T11:36:24.124 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.115+0000 7f25a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:24.124 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.115+0000 7f25a7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:36:24.124 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.115+0000 7f25a7fff640 1 -- 172.21.15.72:0/1123298873 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c01c95b0 con 0x7f25c0071c30 2024-03-29T11:36:24.124 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.115+0000 7f25a7fff640 10 monclient: _finish_auth 0 2024-03-29T11:36:24.124 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.115+0000 7f25a7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:36:24.124 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.115+0000 7f25a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:24.124 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.115+0000 7f25c624a640 5 monclient: authenticate success, global_id 4280 2024-03-29T11:36:24.124 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.115+0000 7f25c624a640 10 monclient: _renew_subs 2024-03-29T11:36:24.124 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.115+0000 7f25c624a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:36:24.124 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.115+0000 7f25c624a640 1 -- 172.21.15.72:0/1123298873 --> v1:172.21.15.72:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25c00b3140 con 0x7f25c0071c30 2024-03-29T11:36:24.125 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.115+0000 7f25c624a640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:24.125 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.115+0000 7f25c624a640 10 monclient: _renew_subs 2024-03-29T11:36:24.125 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.115+0000 7f25c624a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:36:24.125 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.115+0000 7f25c624a640 1 -- 172.21.15.72:0/1123298873 --> v1:172.21.15.72:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f25c01d3270 con 0x7f25c0071c30 2024-03-29T11:36:24.125 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.115+0000 7f25c624a640 1 librados: init done 2024-03-29T11:36:24.125 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.117+0000 7f25a7fff640 1 -- 172.21.15.72:0/1123298873 <== mon.2 v1:172.21.15.72:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25b0004d20 con 0x7f25c0071c30 2024-03-29T11:36:24.125 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.117+0000 7f25a7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:24.125 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.117+0000 7f25bdffb640 4 set_mon_vals no callback set 2024-03-29T11:36:24.125 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.118+0000 7f25a7fff640 1 -- 172.21.15.72:0/1123298873 <== mon.2 v1:172.21.15.72:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f25b0001250 con 0x7f25c0071c30 2024-03-29T11:36:24.125 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.118+0000 7f25a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:24.125 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.118+0000 7f25a7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:24.125 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.118+0000 7f25a7fff640 10 monclient: dump: 2024-03-29T11:36:24.125 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:36:24.125 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:36:24.125 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:24.126 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:24.126 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:24.126 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:24.126 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:36:24.126 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:36:24.126 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:36:24.126 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:24.126 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.118+0000 7f25c624a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:24.126 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.118+0000 7f25c624a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:24.126 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.118+0000 7f25c624a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:36:24.126 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.118+0000 7f25c624a640 1 -- 172.21.15.72:0/1123298873 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25a0005080 con 0x7f25c0071c30 2024-03-29T11:36:24.126 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.118+0000 7f25a7fff640 1 -- 172.21.15.72:0/1123298873 <== mon.2 v1:172.21.15.72:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f25b001ad00 con 0x7f25c0071c30 2024-03-29T11:36:24.126 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.119+0000 7f25beffd640 10 monclient: get_auth_request con 0x7f25ac072150 auth_method 0 2024-03-29T11:36:24.126 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.119+0000 7f25a7fff640 1 -- 172.21.15.72:0/1123298873 <== mon.2 v1:172.21.15.72:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f25b0084b50 con 0x7f25c0071c30 2024-03-29T11:36:24.126 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.119+0000 7f25a7fff640 10 client.4280.objecter ms_dispatch 0x7f25c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:24.127 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.119+0000 7f25a7fff640 3 client.4280.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:24.127 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.119+0000 7f25a7fff640 3 client.4280.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:24.127 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.119+0000 7f25a7fff640 20 client.4280.objecter dump_active .. 0 homeless 2024-03-29T11:36:24.127 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.120+0000 7f25a7fff640 10 client.4280.objecter ms_handle_connect 0x7f25ac072150 2024-03-29T11:36:24.127 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.125+0000 7f25a7fff640 1 -- 172.21.15.72:0/1123298873 <== mon.2 v1:172.21.15.72:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f25b00504f0 con 0x7f25c0071c30 2024-03-29T11:36:24.127 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.125+0000 7f25a7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:24.127 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.125+0000 7f25a7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:24.426 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.428+0000 7f25c624a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:24.427 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.428+0000 7f25c624a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:24.427 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.428+0000 7f25c624a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:36:24.427 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.428+0000 7f25c624a640 1 -- 172.21.15.72:0/1123298873 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25a0005330 con 0x7f25c0071c30 2024-03-29T11:36:24.427 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.428+0000 7f25a7fff640 1 -- 172.21.15.72:0/1123298873 <== mon.2 v1:172.21.15.72:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f25b0052420 con 0x7f25c0071c30 2024-03-29T11:36:24.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.428+0000 7f25a7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:24.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.428+0000 7f25a7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:24.428 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:36:24.428 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:24.431 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.431+0000 7f25c624a640 10 librados: watch_flush enter 2024-03-29T11:36:24.431 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.431+0000 7f25c624a640 10 librados: watch_flush exit 2024-03-29T11:36:24.431 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.431+0000 7f25c624a640 20 client.4280.objecter shutdown clearing up homeless session... 2024-03-29T11:36:24.431 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.431+0000 7f25c624a640 10 client.4280.objecter successfully canceled tick 2024-03-29T11:36:24.431 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.431+0000 7f25c624a640 1 -- 172.21.15.72:0/1123298873 >> v1:172.21.15.72:6816/1578866102 conn(0x7f25ac072150 legacy=0x7f25ac074610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:24.431 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.431+0000 7f25c624a640 10 monclient: shutdown 2024-03-29T11:36:24.431 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.431+0000 7f25c624a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:24.431 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.431+0000 7f25c624a640 1 -- 172.21.15.72:0/1123298873 >> v1:172.21.15.72:6790/0 conn(0x7f25c0071c30 legacy=0x7f25c00b1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:24.431 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.432+0000 7f25c624a640 1 -- 172.21.15.72:0/1123298873 shutdown_connections 2024-03-29T11:36:24.431 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.432+0000 7f25c624a640 1 -- 172.21.15.72:0/1123298873 >> 172.21.15.72:0/1123298873 conn(0x7f25c006b660 msgr2=0x7f25c006ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:24.432 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.432+0000 7f25c624a640 1 -- 172.21.15.72:0/1123298873 shutdown_connections 2024-03-29T11:36:24.432 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.432+0000 7f25c624a640 1 -- 172.21.15.72:0/1123298873 wait complete. 2024-03-29T11:36:24.432 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:24.432+0000 7f25c624a640 1 librados: shutdown 2024-03-29T11:36:24.443 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:31.250 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:31.382 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.382+0000 7f0b23a51640 10 monclient: get_monmap_and_config 2024-03-29T11:36:31.382 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.382+0000 7f0b23a51640 10 monclient: build_initial_monmap 2024-03-29T11:36:31.382 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.382+0000 7f0b23a51640 10 monclient: monmap: 2024-03-29T11:36:31.383 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:36:31.383 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:36:31.383 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:36:31.384517+0000 2024-03-29T11:36:31.383 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:36:31.384517+0000 2024-03-29T11:36:31.383 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:31.383 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:31.383 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:36:31.383 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:36:31.383 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:36:31.383 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:31.383 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.383+0000 7f0b23a51640 1 Processor -- start 2024-03-29T11:36:31.386 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.383+0000 7f0b23a51640 1 -- start start 2024-03-29T11:36:31.386 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.383+0000 7f0b23a51640 10 monclient: init 2024-03-29T11:36:31.386 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.384+0000 7f0b23a51640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:31.386 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.384+0000 7f0b23a51640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:36:31.386 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.384+0000 7f0b23a51640 10 monclient(hunting): picked mon.noname-b con 0x7f0b1c1457f0 addr v1:172.21.15.72:6790/0 2024-03-29T11:36:31.386 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.384+0000 7f0b23a51640 10 monclient(hunting): picked mon.noname-c con 0x7f0b1c13f650 addr v1:172.21.15.89:6789/0 2024-03-29T11:36:31.386 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.384+0000 7f0b23a51640 10 monclient(hunting): picked mon.noname-a con 0x7f0b1c0a9d70 addr v1:172.21.15.72:6789/0 2024-03-29T11:36:31.386 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.384+0000 7f0b23a51640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b1c0aaef0 con 0x7f0b1c0a9d70 2024-03-29T11:36:31.386 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.384+0000 7f0b23a51640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b1c145bd0 con 0x7f0b1c13f650 2024-03-29T11:36:31.386 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.384+0000 7f0b23a51640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b1c145eb0 con 0x7f0b1c1457f0 2024-03-29T11:36:31.386 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.384+0000 7f0b23a51640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:31.386 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.384+0000 7f0b23a51640 10 monclient(hunting): authenticate will time out at 950.673847s 2024-03-29T11:36:31.386 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.384+0000 7f0b217ed640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f0b1c1457f0 0x7f0b1c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:58158/0 (socket says 172.21.15.72:58158) 2024-03-29T11:36:31.387 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.385+0000 7f0b217ed640 1 -- 172.21.15.72:0/3528694393 learned_addr learned my addr 172.21.15.72:0/3528694393 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:36:31.387 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.385+0000 7f0b13fff640 1 -- 172.21.15.72:0/3528694393 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2324426871 0 0) 0x7f0b1c145eb0 con 0x7f0b1c1457f0 2024-03-29T11:36:31.387 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.385+0000 7f0b13fff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:31.387 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.385+0000 7f0b13fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:31.387 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.385+0000 7f0b13fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:31.387 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.385+0000 7f0b13fff640 10 monclient(hunting): my global_id is 4283 2024-03-29T11:36:31.387 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.385+0000 7f0b13fff640 1 -- 172.21.15.72:0/3528694393 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b08003520 con 0x7f0b1c1457f0 2024-03-29T11:36:31.387 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.385+0000 7f0b13fff640 1 -- 172.21.15.72:0/3528694393 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3466352462 0 0) 0x7f0b08003520 con 0x7f0b1c1457f0 2024-03-29T11:36:31.387 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.385+0000 7f0b13fff640 1 -- 172.21.15.72:0/3528694393 >> v1:172.21.15.89:6789/0 conn(0x7f0b1c13f650 legacy=0x7f0b1c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:31.387 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.385+0000 7f0b13fff640 1 -- 172.21.15.72:0/3528694393 >> v1:172.21.15.72:6789/0 conn(0x7f0b1c0a9d70 legacy=0x7f0b1c14d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:36:31.389 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.388+0000 7f0b13fff640 10 monclient: _finish_hunting 0 2024-03-29T11:36:31.390 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.388+0000 7f0b13fff640 1 monclient: found mon.noname-b 2024-03-29T11:36:31.390 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.388+0000 7f0b13fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:31.390 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.388+0000 7f0b13fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.72:6790/0 2024-03-29T11:36:31.390 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.388+0000 7f0b13fff640 1 -- 172.21.15.72:0/3528694393 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b1c146340 con 0x7f0b1c1457f0 2024-03-29T11:36:31.390 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.388+0000 7f0b13fff640 10 monclient: _finish_auth 0 2024-03-29T11:36:31.390 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.388+0000 7f0b13fff640 10 monclient: _check_auth_tickets 2024-03-29T11:36:31.390 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.388+0000 7f0b13fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:31.390 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.388+0000 7f0b13fff640 1 -- 172.21.15.72:0/3528694393 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f0b18001670 con 0x7f0b1c1457f0 2024-03-29T11:36:31.390 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.388+0000 7f0b13fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:31.390 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.388+0000 7f0b13fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:31.390 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.388+0000 7f0b13fff640 10 monclient: dump: 2024-03-29T11:36:31.390 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:36:31.390 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:36:31.390 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:31.391 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:31.391 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:31.391 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:31.391 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:36:31.391 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:36:31.391 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:36:31.391 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:31.391 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.388+0000 7f0b23a51640 5 monclient: authenticate success, global_id 4283 2024-03-29T11:36:31.391 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.388+0000 7f0b23a51640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:31.391 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.389+0000 7f0b13fff640 1 -- 172.21.15.72:0/3528694393 <== mon.2 v1:172.21.15.72:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b180018d0 con 0x7f0b1c1457f0 2024-03-29T11:36:31.391 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.389+0000 7f0b13fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:31.391 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.389+0000 7f0b13fff640 1 -- 172.21.15.72:0/3528694393 <== mon.2 v1:172.21.15.72:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f0b18001d20 con 0x7f0b1c1457f0 2024-03-29T11:36:31.391 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.389+0000 7f0b13fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:31.391 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.389+0000 7f0b13fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:31.391 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.389+0000 7f0b13fff640 10 monclient: dump: 2024-03-29T11:36:31.391 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:36:31.392 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:36:31.392 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:31.392 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:31.392 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:31.392 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:31.392 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:36:31.392 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:36:31.392 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:36:31.392 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:31.392 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.389+0000 7f0b23a51640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:31.392 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.389+0000 7f0b23a51640 4 set_mon_vals no callback set 2024-03-29T11:36:31.392 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.390+0000 7f0b23a51640 10 monclient: shutdown 2024-03-29T11:36:31.392 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.390+0000 7f0b23a51640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:31.392 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.390+0000 7f0b23a51640 1 -- 172.21.15.72:0/3528694393 >> v1:172.21.15.72:6790/0 conn(0x7f0b1c1457f0 legacy=0x7f0b1c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:31.400 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.390+0000 7f0b23a51640 1 -- 172.21.15.72:0/3528694393 shutdown_connections 2024-03-29T11:36:31.400 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.390+0000 7f0b23a51640 1 -- 172.21.15.72:0/3528694393 >> 172.21.15.72:0/3528694393 conn(0x7f0b1c066a00 msgr2=0x7f0b1c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:31.401 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.390+0000 7f0b23a51640 1 -- 172.21.15.72:0/3528694393 shutdown_connections 2024-03-29T11:36:31.401 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.390+0000 7f0b23a51640 1 -- 172.21.15.72:0/3528694393 wait complete. 2024-03-29T11:36:31.401 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.391+0000 7f0b23a51640 10 monclient: build_initial_monmap 2024-03-29T11:36:31.401 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.391+0000 7f0b23a51640 10 monclient: monmap: 2024-03-29T11:36:31.401 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:36:31.401 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:31.401 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:36:31.401 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:36:31.401 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:31.401 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:31.401 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:36:31.401 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:36:31.401 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:36:31.401 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:31.402 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.392+0000 7f0b23a51640 1 librados: starting msgr at 2024-03-29T11:36:31.402 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.392+0000 7f0b23a51640 1 librados: starting objecter 2024-03-29T11:36:31.402 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.392+0000 7f0b23a51640 1 Processor -- start 2024-03-29T11:36:31.402 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.392+0000 7f0b23a51640 1 -- start start 2024-03-29T11:36:31.402 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.392+0000 7f0b23a51640 1 librados: setting wanted keys 2024-03-29T11:36:31.402 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.392+0000 7f0b23a51640 1 librados: calling monclient init 2024-03-29T11:36:31.402 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.392+0000 7f0b23a51640 10 monclient: init 2024-03-29T11:36:31.402 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.393+0000 7f0b23a51640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:31.402 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.393+0000 7f0b23a51640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:36:31.402 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.393+0000 7f0b23a51640 10 monclient(hunting): picked mon.noname-c con 0x7f0b1c0a9d70 addr v1:172.21.15.72:6790/0 2024-03-29T11:36:31.402 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.393+0000 7f0b23a51640 10 monclient(hunting): picked mon.noname-a con 0x7f0b1c13f650 addr v1:172.21.15.72:6789/0 2024-03-29T11:36:31.402 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.393+0000 7f0b23a51640 10 monclient(hunting): picked mon.noname-b con 0x7f0b1c1457f0 addr v1:172.21.15.89:6789/0 2024-03-29T11:36:31.402 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.393+0000 7f0b23a51640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b1c136790 con 0x7f0b1c13f650 2024-03-29T11:36:31.402 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.393+0000 7f0b23a51640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b1c1364d0 con 0x7f0b1c1457f0 2024-03-29T11:36:31.402 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.393+0000 7f0b23a51640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b1c136d20 con 0x7f0b1c0a9d70 2024-03-29T11:36:31.403 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.393+0000 7f0b23a51640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:31.403 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.393+0000 7f0b23a51640 10 monclient(hunting): authenticate will time out at 950.682411s 2024-03-29T11:36:31.403 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.393+0000 7f0b21fee640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f0b1c1457f0 0x7f0b1c137e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:56972/0 (socket says 172.21.15.72:56972) 2024-03-29T11:36:31.403 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.393+0000 7f0b21fee640 1 -- 172.21.15.72:0/2661508691 learned_addr learned my addr 172.21.15.72:0/2661508691 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:36:31.403 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.394+0000 7f0b11ffb640 10 client.?.objecter ms_handle_connect 0x7f0b1c1457f0 2024-03-29T11:36:31.403 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.394+0000 7f0b11ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:31.403 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.394+0000 7f0b11ffb640 10 client.?.objecter ms_handle_connect 0x7f0b1c0a9d70 2024-03-29T11:36:31.403 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.394+0000 7f0b11ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:31.403 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.394+0000 7f0b11ffb640 1 -- 172.21.15.72:0/2661508691 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2957737918 0 0) 0x7f0b1c1364d0 con 0x7f0b1c1457f0 2024-03-29T11:36:31.403 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.394+0000 7f0b11ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:36:31.403 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.394+0000 7f0b11ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:31.403 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.394+0000 7f0b11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:31.403 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.395+0000 7f0b11ffb640 10 monclient(hunting): my global_id is 4282 2024-03-29T11:36:31.403 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.395+0000 7f0b11ffb640 1 -- 172.21.15.72:0/2661508691 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b00004eb0 con 0x7f0b1c1457f0 2024-03-29T11:36:31.403 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.395+0000 7f0b11ffb640 1 -- 172.21.15.72:0/2661508691 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4021366596 0 0) 0x7f0b1c136d20 con 0x7f0b1c0a9d70 2024-03-29T11:36:31.404 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.395+0000 7f0b11ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:36:31.404 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.395+0000 7f0b11ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:31.404 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.395+0000 7f0b11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:31.404 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.395+0000 7f0b11ffb640 10 monclient(hunting): my global_id is 4286 2024-03-29T11:36:31.404 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.395+0000 7f0b11ffb640 1 -- 172.21.15.72:0/2661508691 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b1c1364d0 con 0x7f0b1c0a9d70 2024-03-29T11:36:31.404 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.395+0000 7f0b11ffb640 1 -- 172.21.15.72:0/2661508691 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1759192953 0 0) 0x7f0b00004eb0 con 0x7f0b1c1457f0 2024-03-29T11:36:31.404 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.395+0000 7f0b11ffb640 1 -- 172.21.15.72:0/2661508691 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b1c136d20 con 0x7f0b1c1457f0 2024-03-29T11:36:31.404 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.395+0000 7f0b11ffb640 1 -- 172.21.15.72:0/2661508691 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f0b14001c60 con 0x7f0b1c1457f0 2024-03-29T11:36:31.404 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.395+0000 7f0b11ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:31.404 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.395+0000 7f0b11ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:31.404 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.395+0000 7f0b11ffb640 10 monclient(hunting): dump: 2024-03-29T11:36:31.404 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:36:31.404 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:36:31.404 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:31.404 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:31.404 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:31.405 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:31.405 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:36:31.405 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:36:31.405 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:36:31.405 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:31.405 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.395+0000 7f0b11ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:31.405 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.396+0000 7f0b11ffb640 1 -- 172.21.15.72:0/2661508691 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 788676220 0 0) 0x7f0b1c1364d0 con 0x7f0b1c0a9d70 2024-03-29T11:36:31.405 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.396+0000 7f0b11ffb640 1 -- 172.21.15.72:0/2661508691 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b00004eb0 con 0x7f0b1c0a9d70 2024-03-29T11:36:31.405 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.396+0000 7f0b11ffb640 1 -- 172.21.15.72:0/2661508691 <== mon.1 v1:172.21.15.89:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1061034608 0 0) 0x7f0b1c136d20 con 0x7f0b1c1457f0 2024-03-29T11:36:31.405 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.396+0000 7f0b11ffb640 1 -- 172.21.15.72:0/2661508691 >> v1:172.21.15.72:6790/0 conn(0x7f0b1c0a9d70 legacy=0x7f0b1c070ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:31.405 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.396+0000 7f0b11ffb640 1 -- 172.21.15.72:0/2661508691 >> v1:172.21.15.72:6789/0 conn(0x7f0b1c13f650 legacy=0x7f0b1c137700 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:36:31.405 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.396+0000 7f0b11ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:36:31.405 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.396+0000 7f0b11ffb640 1 monclient: found mon.b 2024-03-29T11:36:31.405 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.396+0000 7f0b11ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:31.405 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.396+0000 7f0b11ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:36:31.406 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.396+0000 7f0b11ffb640 1 -- 172.21.15.72:0/2661508691 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b1c137000 con 0x7f0b1c1457f0 2024-03-29T11:36:31.406 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.397+0000 7f0b11ffb640 10 monclient: _finish_auth 0 2024-03-29T11:36:31.406 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.397+0000 7f0b11ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:36:31.406 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.397+0000 7f0b11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:31.406 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.397+0000 7f0b23a51640 5 monclient: authenticate success, global_id 4282 2024-03-29T11:36:31.406 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.397+0000 7f0b23a51640 10 monclient: _renew_subs 2024-03-29T11:36:31.406 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.397+0000 7f0b23a51640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:36:31.406 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.397+0000 7f0b23a51640 1 -- 172.21.15.72:0/2661508691 --> v1:172.21.15.89:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b1c1372e0 con 0x7f0b1c1457f0 2024-03-29T11:36:31.406 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.397+0000 7f0b23a51640 10 client.4282.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:31.406 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.397+0000 7f0b23a51640 10 monclient: _renew_subs 2024-03-29T11:36:31.406 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.397+0000 7f0b23a51640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:36:31.406 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.397+0000 7f0b23a51640 1 -- 172.21.15.72:0/2661508691 --> v1:172.21.15.89:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b1c1e7520 con 0x7f0b1c1457f0 2024-03-29T11:36:31.406 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.397+0000 7f0b23a51640 1 librados: init done 2024-03-29T11:36:31.406 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.397+0000 7f0b11ffb640 1 -- 172.21.15.72:0/2661508691 <== mon.1 v1:172.21.15.89:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b14001dc0 con 0x7f0b1c1457f0 2024-03-29T11:36:31.406 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.397+0000 7f0b11ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:31.407 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.397+0000 7f0b11ffb640 1 -- 172.21.15.72:0/2661508691 <== mon.1 v1:172.21.15.89:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f0b14003b50 con 0x7f0b1c1457f0 2024-03-29T11:36:31.407 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.397+0000 7f0b11ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:31.407 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.397+0000 7f0b11ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:31.407 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.397+0000 7f0b11ffb640 10 monclient: dump: 2024-03-29T11:36:31.407 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:36:31.407 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:36:31.407 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:31.407 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:31.407 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:31.407 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:31.407 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:36:31.407 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:36:31.407 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:36:31.407 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:31.407 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.397+0000 7f0b137fe640 4 set_mon_vals no callback set 2024-03-29T11:36:31.408 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.398+0000 7f0b23a51640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:31.408 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.398+0000 7f0b23a51640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:31.408 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.398+0000 7f0b23a51640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:36:31.408 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.398+0000 7f0b23a51640 1 -- 172.21.15.72:0/2661508691 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0afc005080 con 0x7f0b1c1457f0 2024-03-29T11:36:31.409 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.402+0000 7f0b11ffb640 1 -- 172.21.15.72:0/2661508691 <== mon.1 v1:172.21.15.89:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f0b14003cb0 con 0x7f0b1c1457f0 2024-03-29T11:36:31.409 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.402+0000 7f0b11ffb640 1 -- 172.21.15.72:0/2661508691 <== mon.1 v1:172.21.15.89:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f0b14084c40 con 0x7f0b1c1457f0 2024-03-29T11:36:31.409 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.402+0000 7f0b217ed640 10 monclient: get_auth_request con 0x7f0b0006df90 auth_method 0 2024-03-29T11:36:31.409 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.402+0000 7f0b11ffb640 10 client.4282.objecter ms_dispatch 0x7f0b1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:31.409 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.402+0000 7f0b11ffb640 3 client.4282.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:31.409 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.403+0000 7f0b11ffb640 3 client.4282.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:31.409 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.403+0000 7f0b11ffb640 20 client.4282.objecter dump_active .. 0 homeless 2024-03-29T11:36:31.410 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.403+0000 7f0b11ffb640 10 client.4282.objecter ms_handle_connect 0x7f0b0006df90 2024-03-29T11:36:31.410 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.408+0000 7f0b11ffb640 1 -- 172.21.15.72:0/2661508691 <== mon.1 v1:172.21.15.89:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0b14050350 con 0x7f0b1c1457f0 2024-03-29T11:36:31.410 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.408+0000 7f0b11ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:31.410 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.408+0000 7f0b11ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:31.717 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.717+0000 7f0b23a51640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:31.717 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.717+0000 7f0b23a51640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:31.717 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.717+0000 7f0b23a51640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:36:31.718 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.717+0000 7f0b23a51640 1 -- 172.21.15.72:0/2661508691 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0afc005330 con 0x7f0b1c1457f0 2024-03-29T11:36:31.718 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.718+0000 7f0b11ffb640 1 -- 172.21.15.72:0/2661508691 <== mon.1 v1:172.21.15.89:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f0b14052450 con 0x7f0b1c1457f0 2024-03-29T11:36:31.718 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.718+0000 7f0b11ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:31.718 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.718+0000 7f0b11ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:31.719 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:36:31.719 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:31.722 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.721+0000 7f0b23a51640 10 librados: watch_flush enter 2024-03-29T11:36:31.722 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.721+0000 7f0b23a51640 10 librados: watch_flush exit 2024-03-29T11:36:31.722 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.721+0000 7f0b23a51640 20 client.4282.objecter shutdown clearing up homeless session... 2024-03-29T11:36:31.722 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.721+0000 7f0b23a51640 10 client.4282.objecter successfully canceled tick 2024-03-29T11:36:31.722 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.722+0000 7f0b23a51640 1 -- 172.21.15.72:0/2661508691 >> v1:172.21.15.72:6816/1578866102 conn(0x7f0b0006df90 legacy=0x7f0b00070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:31.722 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.722+0000 7f0b23a51640 10 monclient: shutdown 2024-03-29T11:36:31.722 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.722+0000 7f0b23a51640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:31.722 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.722+0000 7f0b23a51640 1 -- 172.21.15.72:0/2661508691 >> v1:172.21.15.89:6789/0 conn(0x7f0b1c1457f0 legacy=0x7f0b1c137e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:31.723 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.722+0000 7f0b23a51640 1 -- 172.21.15.72:0/2661508691 shutdown_connections 2024-03-29T11:36:31.723 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.722+0000 7f0b23a51640 1 -- 172.21.15.72:0/2661508691 >> 172.21.15.72:0/2661508691 conn(0x7f0b1c066a00 msgr2=0x7f0b1c0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:31.723 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.722+0000 7f0b23a51640 1 -- 172.21.15.72:0/2661508691 shutdown_connections 2024-03-29T11:36:31.723 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.722+0000 7f0b23a51640 1 -- 172.21.15.72:0/2661508691 wait complete. 2024-03-29T11:36:31.723 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:31.722+0000 7f0b23a51640 1 librados: shutdown 2024-03-29T11:36:31.737 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:38.545 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:38.691 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.691+0000 7f666ba0f640 10 monclient: get_monmap_and_config 2024-03-29T11:36:38.691 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.691+0000 7f666ba0f640 10 monclient: build_initial_monmap 2024-03-29T11:36:38.691 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.692+0000 7f666ba0f640 10 monclient: monmap: 2024-03-29T11:36:38.691 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:36:38.692 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:36:38.692 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:36:38.692976+0000 2024-03-29T11:36:38.692 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:36:38.692976+0000 2024-03-29T11:36:38.692 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:38.692 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:38.692 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:36:38.692 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:36:38.692 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:36:38.692 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:38.693 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.693+0000 7f666ba0f640 1 Processor -- start 2024-03-29T11:36:38.693 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.693+0000 7f666ba0f640 1 -- start start 2024-03-29T11:36:38.693 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.693+0000 7f666ba0f640 10 monclient: init 2024-03-29T11:36:38.693 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.694+0000 7f666ba0f640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:38.693 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.694+0000 7f666ba0f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:36:38.695 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.695+0000 7f666ba0f640 10 monclient(hunting): picked mon.noname-c con 0x7f66640a9750 addr v1:172.21.15.89:6789/0 2024-03-29T11:36:38.695 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.695+0000 7f666ba0f640 10 monclient(hunting): picked mon.noname-b con 0x7f666413e1e0 addr v1:172.21.15.72:6790/0 2024-03-29T11:36:38.695 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.695+0000 7f666ba0f640 10 monclient(hunting): picked mon.noname-a con 0x7f6664071c30 addr v1:172.21.15.72:6789/0 2024-03-29T11:36:38.695 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.695+0000 7f666ba0f640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66640ab300 con 0x7f6664071c30 2024-03-29T11:36:38.695 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.695+0000 7f666ba0f640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66640ac2f0 con 0x7f66640a9750 2024-03-29T11:36:38.695 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.695+0000 7f666ba0f640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66640ad8d0 con 0x7f666413e1e0 2024-03-29T11:36:38.695 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.695+0000 7f666ba0f640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:38.695 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.695+0000 7f666ba0f640 10 monclient(hunting): authenticate will time out at 957.983679s 2024-03-29T11:36:38.697 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.698+0000 7f6669fac640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f6664071c30 0x7f66640ace20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60459/0 (socket says 172.21.15.72:60459) 2024-03-29T11:36:38.697 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.698+0000 7f6669fac640 1 -- 172.21.15.72:0/2749537313 learned_addr learned my addr 172.21.15.72:0/2749537313 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:36:38.699 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.699+0000 7f665bfff640 1 -- 172.21.15.72:0/2749537313 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2018765993 0 0) 0x7f66640ab300 con 0x7f6664071c30 2024-03-29T11:36:38.699 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.699+0000 7f665bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:38.699 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.699+0000 7f665bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:38.699 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.699+0000 7f665bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:38.699 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.699+0000 7f665bfff640 10 monclient(hunting): my global_id is 4278 2024-03-29T11:36:38.699 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.699+0000 7f665bfff640 1 -- 172.21.15.72:0/2749537313 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6650003520 con 0x7f6664071c30 2024-03-29T11:36:38.699 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.700+0000 7f665bfff640 1 -- 172.21.15.72:0/2749537313 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4099985124 0 0) 0x7f6650003520 con 0x7f6664071c30 2024-03-29T11:36:38.699 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.700+0000 7f665bfff640 1 -- 172.21.15.72:0/2749537313 >> v1:172.21.15.72:6790/0 conn(0x7f666413e1e0 legacy=0x7f66640ac710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:36:38.702 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.702+0000 7f665bfff640 1 -- 172.21.15.72:0/2749537313 >> v1:172.21.15.89:6789/0 conn(0x7f66640a9750 legacy=0x7f666413dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:38.702 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.702+0000 7f665bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:36:38.702 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.702+0000 7f665bfff640 1 monclient: found mon.noname-a 2024-03-29T11:36:38.702 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.702+0000 7f665bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:38.702 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.702+0000 7f665bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.72:6789/0 2024-03-29T11:36:38.702 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.702+0000 7f665bfff640 1 -- 172.21.15.72:0/2749537313 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66640ada60 con 0x7f6664071c30 2024-03-29T11:36:38.703 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.702+0000 7f665bfff640 10 monclient: _finish_auth 0 2024-03-29T11:36:38.703 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.702+0000 7f665bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:36:38.703 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.702+0000 7f665bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:38.703 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.702+0000 7f665bfff640 1 -- 172.21.15.72:0/2749537313 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f665c0019f0 con 0x7f6664071c30 2024-03-29T11:36:38.703 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.702+0000 7f665bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:38.703 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.702+0000 7f665bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:38.703 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.702+0000 7f665bfff640 10 monclient: dump: 2024-03-29T11:36:38.703 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:36:38.703 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:36:38.703 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:38.703 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:38.704 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:38.704 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:38.704 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:36:38.704 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:36:38.704 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:36:38.704 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:38.704 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.702+0000 7f666ba0f640 5 monclient: authenticate success, global_id 4278 2024-03-29T11:36:38.704 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.702+0000 7f666ba0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:38.704 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.705+0000 7f665bfff640 1 -- 172.21.15.72:0/2749537313 <== mon.0 v1:172.21.15.72:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f665c001bc0 con 0x7f6664071c30 2024-03-29T11:36:38.704 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.705+0000 7f665bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:38.705 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.705+0000 7f665bfff640 1 -- 172.21.15.72:0/2749537313 <== mon.0 v1:172.21.15.72:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f665c001fd0 con 0x7f6664071c30 2024-03-29T11:36:38.705 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.705+0000 7f665bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:38.705 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.705+0000 7f665bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:38.705 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.705+0000 7f665bfff640 10 monclient: dump: 2024-03-29T11:36:38.705 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:36:38.705 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:36:38.705 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:38.705 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:38.705 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:38.706 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:38.706 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:36:38.706 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:36:38.706 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:36:38.706 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:38.706 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.705+0000 7f666ba0f640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:38.706 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.705+0000 7f666ba0f640 4 set_mon_vals no callback set 2024-03-29T11:36:38.707 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.706+0000 7f666ba0f640 10 monclient: shutdown 2024-03-29T11:36:38.707 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.706+0000 7f666ba0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:38.707 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.706+0000 7f666ba0f640 1 -- 172.21.15.72:0/2749537313 >> v1:172.21.15.72:6789/0 conn(0x7f6664071c30 legacy=0x7f66640ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:38.707 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.707+0000 7f666ba0f640 1 -- 172.21.15.72:0/2749537313 shutdown_connections 2024-03-29T11:36:38.707 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.707+0000 7f666ba0f640 1 -- 172.21.15.72:0/2749537313 >> 172.21.15.72:0/2749537313 conn(0x7f666406b660 msgr2=0x7f666406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:38.708 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.707+0000 7f666ba0f640 1 -- 172.21.15.72:0/2749537313 shutdown_connections 2024-03-29T11:36:38.708 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.707+0000 7f666ba0f640 1 -- 172.21.15.72:0/2749537313 wait complete. 2024-03-29T11:36:38.708 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.707+0000 7f666ba0f640 10 monclient: build_initial_monmap 2024-03-29T11:36:38.708 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.707+0000 7f666ba0f640 10 monclient: monmap: 2024-03-29T11:36:38.708 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:36:38.708 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:38.708 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:36:38.708 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:36:38.709 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:38.709 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:38.709 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:36:38.709 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:36:38.709 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:36:38.709 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:38.709 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.708+0000 7f666ba0f640 1 librados: starting msgr at 2024-03-29T11:36:38.709 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.708+0000 7f666ba0f640 1 librados: starting objecter 2024-03-29T11:36:38.709 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.708+0000 7f666ba0f640 1 Processor -- start 2024-03-29T11:36:38.711 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.711+0000 7f666ba0f640 1 -- start start 2024-03-29T11:36:38.711 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.711+0000 7f666ba0f640 1 librados: setting wanted keys 2024-03-29T11:36:38.711 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.711+0000 7f666ba0f640 1 librados: calling monclient init 2024-03-29T11:36:38.711 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.711+0000 7f666ba0f640 10 monclient: init 2024-03-29T11:36:38.712 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.712+0000 7f666ba0f640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:38.712 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.712+0000 7f666ba0f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:36:38.712 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.712+0000 7f666ba0f640 10 monclient(hunting): picked mon.noname-b con 0x7f6664071c30 addr v1:172.21.15.89:6789/0 2024-03-29T11:36:38.712 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.712+0000 7f666ba0f640 10 monclient(hunting): picked mon.noname-c con 0x7f66640a9750 addr v1:172.21.15.72:6790/0 2024-03-29T11:36:38.713 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.712+0000 7f666ba0f640 10 monclient(hunting): picked mon.noname-a con 0x7f666413e1e0 addr v1:172.21.15.72:6789/0 2024-03-29T11:36:38.713 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.712+0000 7f666ba0f640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6664140930 con 0x7f666413e1e0 2024-03-29T11:36:38.713 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.712+0000 7f666ba0f640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6664140c10 con 0x7f6664071c30 2024-03-29T11:36:38.713 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.712+0000 7f666ba0f640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6664140ef0 con 0x7f66640a9750 2024-03-29T11:36:38.713 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.712+0000 7f666ba0f640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:38.713 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.712+0000 7f666ba0f640 10 monclient(hunting): authenticate will time out at 958.000441s 2024-03-29T11:36:38.716 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.717+0000 7f66697ab640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f6664071c30 0x7f66640b1400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:32983/0 (socket says 172.21.15.72:32983) 2024-03-29T11:36:38.717 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.717+0000 7f66697ab640 1 -- 172.21.15.72:0/4292540197 learned_addr learned my addr 172.21.15.72:0/4292540197 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:36:38.718 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.718+0000 7f6659ffb640 10 client.?.objecter ms_handle_connect 0x7f66640a9750 2024-03-29T11:36:38.718 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.718+0000 7f6659ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:38.718 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.719+0000 7f6659ffb640 10 client.?.objecter ms_handle_connect 0x7f6664071c30 2024-03-29T11:36:38.718 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.719+0000 7f6659ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:38.718 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.719+0000 7f6659ffb640 1 -- 172.21.15.72:0/4292540197 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1392981981 0 0) 0x7f6664140ef0 con 0x7f66640a9750 2024-03-29T11:36:38.718 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.719+0000 7f6659ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:36:38.718 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.719+0000 7f6659ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:38.718 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.719+0000 7f6659ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:38.718 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.719+0000 7f6659ffb640 10 monclient(hunting): my global_id is 4289 2024-03-29T11:36:38.718 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.719+0000 7f6659ffb640 1 -- 172.21.15.72:0/4292540197 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6648004eb0 con 0x7f66640a9750 2024-03-29T11:36:38.719 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.719+0000 7f6659ffb640 1 -- 172.21.15.72:0/4292540197 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2174929338 0 0) 0x7f6648004eb0 con 0x7f66640a9750 2024-03-29T11:36:38.719 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.719+0000 7f6659ffb640 1 -- 172.21.15.72:0/4292540197 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6664140ef0 con 0x7f66640a9750 2024-03-29T11:36:38.719 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.719+0000 7f6659ffb640 1 -- 172.21.15.72:0/4292540197 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f66540017d0 con 0x7f66640a9750 2024-03-29T11:36:38.719 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.719+0000 7f6659ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:38.719 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.719+0000 7f6659ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:38.719 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.719+0000 7f6659ffb640 10 monclient(hunting): dump: 2024-03-29T11:36:38.719 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:36:38.719 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:36:38.720 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:38.720 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:38.720 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:38.720 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:38.720 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:36:38.720 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:36:38.720 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:36:38.720 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:38.720 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.719+0000 7f6659ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:38.720 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.719+0000 7f6659ffb640 1 -- 172.21.15.72:0/4292540197 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 935160099 0 0) 0x7f6664140c10 con 0x7f6664071c30 2024-03-29T11:36:38.720 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.719+0000 7f6659ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:36:38.720 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.719+0000 7f6659ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:38.720 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.719+0000 7f6659ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:38.720 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.719+0000 7f6659ffb640 10 monclient(hunting): my global_id is 4288 2024-03-29T11:36:38.720 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.719+0000 7f6659ffb640 1 -- 172.21.15.72:0/4292540197 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6648004eb0 con 0x7f6664071c30 2024-03-29T11:36:38.721 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.719+0000 7f6659ffb640 1 -- 172.21.15.72:0/4292540197 <== mon.2 v1:172.21.15.72:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 385957053 0 0) 0x7f6664140ef0 con 0x7f66640a9750 2024-03-29T11:36:38.721 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.719+0000 7f6659ffb640 1 -- 172.21.15.72:0/4292540197 >> v1:172.21.15.89:6789/0 conn(0x7f6664071c30 legacy=0x7f66640b1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:38.721 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.719+0000 7f6659ffb640 1 -- 172.21.15.72:0/4292540197 >> v1:172.21.15.72:6789/0 conn(0x7f666413e1e0 legacy=0x7f6664141d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:38.721 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.719+0000 7f6659ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:36:38.721 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.719+0000 7f6659ffb640 1 monclient: found mon.c 2024-03-29T11:36:38.721 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.719+0000 7f6659ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:38.721 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.719+0000 7f6659ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:36:38.721 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.719+0000 7f6659ffb640 1 -- 172.21.15.72:0/4292540197 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66640b31e0 con 0x7f66640a9750 2024-03-29T11:36:38.721 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.720+0000 7f6659ffb640 10 monclient: _finish_auth 0 2024-03-29T11:36:38.721 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.720+0000 7f6659ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:36:38.721 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.720+0000 7f6659ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:38.721 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.720+0000 7f6659ffb640 1 -- 172.21.15.72:0/4292540197 <== mon.2 v1:172.21.15.72:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6654001930 con 0x7f66640a9750 2024-03-29T11:36:38.721 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.720+0000 7f6659ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:38.721 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.720+0000 7f6659ffb640 1 -- 172.21.15.72:0/4292540197 <== mon.2 v1:172.21.15.72:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f6654003790 con 0x7f66640a9750 2024-03-29T11:36:38.722 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.720+0000 7f6659ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:38.722 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.720+0000 7f6659ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:38.722 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.720+0000 7f6659ffb640 10 monclient: dump: 2024-03-29T11:36:38.722 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:36:38.722 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:36:38.722 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:38.722 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:38.722 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:38.722 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:38.722 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:36:38.722 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:36:38.722 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:36:38.722 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:38.722 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.720+0000 7f665b7fe640 4 set_mon_vals no callback set 2024-03-29T11:36:38.723 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.721+0000 7f666ba0f640 5 monclient: authenticate success, global_id 4289 2024-03-29T11:36:38.723 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.721+0000 7f666ba0f640 10 monclient: _renew_subs 2024-03-29T11:36:38.723 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.721+0000 7f666ba0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:36:38.724 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.721+0000 7f666ba0f640 1 -- 172.21.15.72:0/4292540197 --> v1:172.21.15.72:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f666413f970 con 0x7f66640a9750 2024-03-29T11:36:38.724 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.722+0000 7f666ba0f640 10 client.4289.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:38.724 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.722+0000 7f666ba0f640 10 monclient: _renew_subs 2024-03-29T11:36:38.724 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.722+0000 7f666ba0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:36:38.724 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.722+0000 7f666ba0f640 1 -- 172.21.15.72:0/4292540197 --> v1:172.21.15.72:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f66641425f0 con 0x7f66640a9750 2024-03-29T11:36:38.724 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.722+0000 7f666ba0f640 1 librados: init done 2024-03-29T11:36:38.724 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.724+0000 7f6659ffb640 1 -- 172.21.15.72:0/4292540197 <== mon.2 v1:172.21.15.72:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f6654019760 con 0x7f66640a9750 2024-03-29T11:36:38.725 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.725+0000 7f6659ffb640 1 -- 172.21.15.72:0/4292540197 <== mon.2 v1:172.21.15.72:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f6654083770 con 0x7f66640a9750 2024-03-29T11:36:38.725 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.725+0000 7f6659ffb640 10 client.4289.objecter ms_dispatch 0x7f6664000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:38.725 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.725+0000 7f6659ffb640 3 client.4289.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:38.725 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.725+0000 7f6659ffb640 3 client.4289.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:38.725 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.725+0000 7f6659ffb640 20 client.4289.objecter dump_active .. 0 homeless 2024-03-29T11:36:38.725 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.725+0000 7f666ba0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:38.725 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.726+0000 7f666ba0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:38.726 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.726+0000 7f666ba0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:36:38.726 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.726+0000 7f666ba0f640 1 -- 172.21.15.72:0/4292540197 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6664142890 con 0x7f66640a9750 2024-03-29T11:36:38.726 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.726+0000 7f66697ab640 10 monclient: get_auth_request con 0x7f664806dcb0 auth_method 0 2024-03-29T11:36:38.726 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.727+0000 7f6659ffb640 10 client.4289.objecter ms_handle_connect 0x7f664806dcb0 2024-03-29T11:36:38.736 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.738+0000 7f6659ffb640 1 -- 172.21.15.72:0/4292540197 <== mon.2 v1:172.21.15.72:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f665404f050 con 0x7f66640a9750 2024-03-29T11:36:38.737 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.738+0000 7f6659ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:38.737 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:38.738+0000 7f6659ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:39.056 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:39.057+0000 7f666ba0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:39.056 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:39.057+0000 7f666ba0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:39.057 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:39.057+0000 7f666ba0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:36:39.057 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:39.057+0000 7f666ba0f640 1 -- 172.21.15.72:0/4292540197 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f66640a9b30 con 0x7f66640a9750 2024-03-29T11:36:39.059 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:39.060+0000 7f6659ffb640 1 -- 172.21.15.72:0/4292540197 <== mon.2 v1:172.21.15.72:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f6654056ba0 con 0x7f66640a9750 2024-03-29T11:36:39.059 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:39.060+0000 7f6659ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:39.059 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:39.060+0000 7f6659ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:39.060 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:36:39.060 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:39.065 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:39.066+0000 7f666ba0f640 10 librados: watch_flush enter 2024-03-29T11:36:39.065 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:39.066+0000 7f666ba0f640 10 librados: watch_flush exit 2024-03-29T11:36:39.065 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:39.066+0000 7f666ba0f640 20 client.4289.objecter shutdown clearing up homeless session... 2024-03-29T11:36:39.065 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:39.066+0000 7f666ba0f640 10 client.4289.objecter successfully canceled tick 2024-03-29T11:36:39.066 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:39.067+0000 7f666ba0f640 1 -- 172.21.15.72:0/4292540197 >> v1:172.21.15.72:6816/1578866102 conn(0x7f664806dcb0 legacy=0x7f6648070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:39.066 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:39.067+0000 7f666ba0f640 10 monclient: shutdown 2024-03-29T11:36:39.066 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:39.067+0000 7f666ba0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:39.066 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:39.067+0000 7f666ba0f640 1 -- 172.21.15.72:0/4292540197 >> v1:172.21.15.72:6790/0 conn(0x7f66640a9750 legacy=0x7f66641415f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:39.069 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:39.070+0000 7f666ba0f640 1 -- 172.21.15.72:0/4292540197 shutdown_connections 2024-03-29T11:36:39.069 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:39.070+0000 7f666ba0f640 1 -- 172.21.15.72:0/4292540197 >> 172.21.15.72:0/4292540197 conn(0x7f666406b660 msgr2=0x7f666413a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:39.070 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:39.070+0000 7f666ba0f640 1 -- 172.21.15.72:0/4292540197 shutdown_connections 2024-03-29T11:36:39.070 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:39.070+0000 7f666ba0f640 1 -- 172.21.15.72:0/4292540197 wait complete. 2024-03-29T11:36:39.070 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:39.071+0000 7f666ba0f640 1 librados: shutdown 2024-03-29T11:36:39.089 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:45.895 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:46.050 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.050+0000 7fe53a225640 10 monclient: get_monmap_and_config 2024-03-29T11:36:46.050 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.050+0000 7fe53a225640 10 monclient: build_initial_monmap 2024-03-29T11:36:46.050 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.050+0000 7fe53a225640 10 monclient: monmap: 2024-03-29T11:36:46.050 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:36:46.050 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:36:46.050 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:36:46.051775+0000 2024-03-29T11:36:46.050 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:36:46.051775+0000 2024-03-29T11:36:46.050 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:46.050 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:46.051 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:36:46.051 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:36:46.051 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:36:46.051 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:46.052 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.053+0000 7fe53a225640 1 Processor -- start 2024-03-29T11:36:46.052 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.053+0000 7fe53a225640 1 -- start start 2024-03-29T11:36:46.052 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.053+0000 7fe53a225640 10 monclient: init 2024-03-29T11:36:46.052 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.053+0000 7fe53a225640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:46.053 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.054+0000 7fe53a225640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:36:46.053 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.054+0000 7fe53a225640 10 monclient(hunting): picked mon.noname-b con 0x7fe5341457f0 addr v1:172.21.15.72:6790/0 2024-03-29T11:36:46.053 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.054+0000 7fe53a225640 10 monclient(hunting): picked mon.noname-a con 0x7fe53413f650 addr v1:172.21.15.72:6789/0 2024-03-29T11:36:46.053 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.054+0000 7fe53a225640 10 monclient(hunting): picked mon.noname-c con 0x7fe5340a9d70 addr v1:172.21.15.89:6789/0 2024-03-29T11:36:46.053 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.054+0000 7fe53a225640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5340aaef0 con 0x7fe53413f650 2024-03-29T11:36:46.053 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.054+0000 7fe53a225640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe534145bd0 con 0x7fe5340a9d70 2024-03-29T11:36:46.053 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.054+0000 7fe53a225640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe534145eb0 con 0x7fe5341457f0 2024-03-29T11:36:46.053 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.054+0000 7fe53a225640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:46.053 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.054+0000 7fe53a225640 10 monclient(hunting): authenticate will time out at 965.342099s 2024-03-29T11:36:46.058 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.059+0000 7fe532ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fe53413f650 0x7fe5340b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49519/0 (socket says 172.21.15.72:49519) 2024-03-29T11:36:46.058 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.059+0000 7fe532ffd640 1 -- 172.21.15.72:0/1329017719 learned_addr learned my addr 172.21.15.72:0/1329017719 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:36:46.060 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.059+0000 7fe5327fc640 1 -- 172.21.15.72:0/1329017719 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 566013883 0 0) 0x7fe534145eb0 con 0x7fe5341457f0 2024-03-29T11:36:46.060 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.060+0000 7fe5327fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:46.060 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.060+0000 7fe5327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:46.060 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.060+0000 7fe5327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:46.060 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.060+0000 7fe5327fc640 10 monclient(hunting): my global_id is 4292 2024-03-29T11:36:46.060 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.060+0000 7fe5327fc640 1 -- 172.21.15.72:0/1329017719 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe51c003520 con 0x7fe5341457f0 2024-03-29T11:36:46.060 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.060+0000 7fe5327fc640 1 -- 172.21.15.72:0/1329017719 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3591055169 0 0) 0x7fe51c003520 con 0x7fe5341457f0 2024-03-29T11:36:46.060 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.060+0000 7fe5327fc640 1 -- 172.21.15.72:0/1329017719 >> v1:172.21.15.89:6789/0 conn(0x7fe5340a9d70 legacy=0x7fe53414d8c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:36:46.060 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.060+0000 7fe5327fc640 1 -- 172.21.15.72:0/1329017719 >> v1:172.21.15.72:6789/0 conn(0x7fe53413f650 legacy=0x7fe5340b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:46.060 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.060+0000 7fe5327fc640 10 monclient: _finish_hunting 0 2024-03-29T11:36:46.060 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.060+0000 7fe5327fc640 1 monclient: found mon.noname-b 2024-03-29T11:36:46.060 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.060+0000 7fe5327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:46.060 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.060+0000 7fe5327fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.72:6790/0 2024-03-29T11:36:46.060 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.060+0000 7fe5327fc640 1 -- 172.21.15.72:0/1329017719 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe534146340 con 0x7fe5341457f0 2024-03-29T11:36:46.061 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.060+0000 7fe5327fc640 10 monclient: _finish_auth 0 2024-03-29T11:36:46.061 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.060+0000 7fe5327fc640 10 monclient: _check_auth_tickets 2024-03-29T11:36:46.061 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.060+0000 7fe5327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:46.061 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.060+0000 7fe5327fc640 1 -- 172.21.15.72:0/1329017719 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fe524001690 con 0x7fe5341457f0 2024-03-29T11:36:46.061 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.060+0000 7fe5327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:46.061 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.060+0000 7fe5327fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:46.061 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.060+0000 7fe5327fc640 10 monclient: dump: 2024-03-29T11:36:46.061 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:36:46.061 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:36:46.061 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:46.061 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:46.061 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:46.061 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:46.061 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:36:46.061 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:36:46.061 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:36:46.062 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:46.062 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.060+0000 7fe5327fc640 1 -- 172.21.15.72:0/1329017719 <== mon.2 v1:172.21.15.72:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe524001860 con 0x7fe5341457f0 2024-03-29T11:36:46.062 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.060+0000 7fe5327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:46.062 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.060+0000 7fe5327fc640 1 -- 172.21.15.72:0/1329017719 <== mon.2 v1:172.21.15.72:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fe524001cb0 con 0x7fe5341457f0 2024-03-29T11:36:46.062 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.060+0000 7fe5327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:46.062 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.060+0000 7fe5327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:46.062 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.060+0000 7fe5327fc640 10 monclient: dump: 2024-03-29T11:36:46.062 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:36:46.062 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:36:46.062 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:46.062 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:46.062 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:46.062 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:46.062 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:36:46.062 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:36:46.063 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:36:46.063 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:46.064 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.061+0000 7fe53a225640 5 monclient: authenticate success, global_id 4292 2024-03-29T11:36:46.064 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.061+0000 7fe53a225640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:46.064 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.061+0000 7fe53a225640 4 set_mon_vals no callback set 2024-03-29T11:36:46.064 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.062+0000 7fe53a225640 10 monclient: shutdown 2024-03-29T11:36:46.064 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.062+0000 7fe53a225640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:46.064 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.062+0000 7fe53a225640 1 -- 172.21.15.72:0/1329017719 >> v1:172.21.15.72:6790/0 conn(0x7fe5341457f0 legacy=0x7fe53413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:46.064 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.062+0000 7fe53a225640 1 -- 172.21.15.72:0/1329017719 shutdown_connections 2024-03-29T11:36:46.064 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.062+0000 7fe53a225640 1 -- 172.21.15.72:0/1329017719 >> 172.21.15.72:0/1329017719 conn(0x7fe534066a00 msgr2=0x7fe534066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:46.064 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.062+0000 7fe53a225640 1 -- 172.21.15.72:0/1329017719 shutdown_connections 2024-03-29T11:36:46.064 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.062+0000 7fe53a225640 1 -- 172.21.15.72:0/1329017719 wait complete. 2024-03-29T11:36:46.064 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.063+0000 7fe53a225640 10 monclient: build_initial_monmap 2024-03-29T11:36:46.064 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.063+0000 7fe53a225640 10 monclient: monmap: 2024-03-29T11:36:46.065 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:36:46.065 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:46.065 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:36:46.065 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:36:46.065 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:46.065 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:46.065 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:36:46.065 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:36:46.065 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:36:46.065 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:46.065 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.064+0000 7fe53a225640 1 librados: starting msgr at 2024-03-29T11:36:46.065 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.064+0000 7fe53a225640 1 librados: starting objecter 2024-03-29T11:36:46.065 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.064+0000 7fe53a225640 1 Processor -- start 2024-03-29T11:36:46.067 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.064+0000 7fe53a225640 1 -- start start 2024-03-29T11:36:46.067 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.064+0000 7fe53a225640 1 librados: setting wanted keys 2024-03-29T11:36:46.067 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.064+0000 7fe53a225640 1 librados: calling monclient init 2024-03-29T11:36:46.067 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.064+0000 7fe53a225640 10 monclient: init 2024-03-29T11:36:46.067 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.065+0000 7fe53a225640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:46.067 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.065+0000 7fe53a225640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:36:46.067 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.065+0000 7fe53a225640 10 monclient(hunting): picked mon.noname-c con 0x7fe5340a9d70 addr v1:172.21.15.72:6790/0 2024-03-29T11:36:46.067 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.065+0000 7fe53a225640 10 monclient(hunting): picked mon.noname-b con 0x7fe53413f650 addr v1:172.21.15.89:6789/0 2024-03-29T11:36:46.067 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.065+0000 7fe53a225640 10 monclient(hunting): picked mon.noname-a con 0x7fe5341457f0 addr v1:172.21.15.72:6789/0 2024-03-29T11:36:46.067 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.065+0000 7fe53a225640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe534142ba0 con 0x7fe5341457f0 2024-03-29T11:36:46.068 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.065+0000 7fe53a225640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe534142df0 con 0x7fe53413f650 2024-03-29T11:36:46.068 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.065+0000 7fe53a225640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe53406df30 con 0x7fe5340a9d70 2024-03-29T11:36:46.068 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.065+0000 7fe53a225640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:46.068 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.065+0000 7fe53a225640 10 monclient(hunting): authenticate will time out at 965.353812s 2024-03-29T11:36:46.069 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.070+0000 7fe532ffd640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe53413f650 0x7fe534144870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:45465/0 (socket says 172.21.15.72:45465) 2024-03-29T11:36:46.069 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.070+0000 7fe532ffd640 1 -- 172.21.15.72:0/2554278530 learned_addr learned my addr 172.21.15.72:0/2554278530 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:36:46.070 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.070+0000 7fe51bfff640 10 client.?.objecter ms_handle_connect 0x7fe53413f650 2024-03-29T11:36:46.070 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.071+0000 7fe51bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:46.073 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.074+0000 7fe51bfff640 10 client.?.objecter ms_handle_connect 0x7fe5341457f0 2024-03-29T11:36:46.073 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.074+0000 7fe51bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:46.079 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.075+0000 7fe51bfff640 1 -- 172.21.15.72:0/2554278530 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3755369764 0 0) 0x7fe534142ba0 con 0x7fe5341457f0 2024-03-29T11:36:46.079 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.075+0000 7fe51bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:46.079 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.075+0000 7fe51bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:46.079 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.075+0000 7fe51bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:46.079 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.075+0000 7fe51bfff640 10 monclient(hunting): my global_id is 4284 2024-03-29T11:36:46.079 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.075+0000 7fe51bfff640 1 -- 172.21.15.72:0/2554278530 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe520004eb0 con 0x7fe5341457f0 2024-03-29T11:36:46.079 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.075+0000 7fe51bfff640 10 client.?.objecter ms_handle_connect 0x7fe5340a9d70 2024-03-29T11:36:46.079 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.075+0000 7fe51bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:46.079 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.075+0000 7fe51bfff640 1 -- 172.21.15.72:0/2554278530 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3675181655 0 0) 0x7fe534142df0 con 0x7fe53413f650 2024-03-29T11:36:46.079 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.076+0000 7fe51bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:46.079 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.076+0000 7fe51bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:46.079 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.076+0000 7fe51bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:46.079 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.076+0000 7fe51bfff640 10 monclient(hunting): my global_id is 4291 2024-03-29T11:36:46.079 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.076+0000 7fe51bfff640 1 -- 172.21.15.72:0/2554278530 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe534142ba0 con 0x7fe53413f650 2024-03-29T11:36:46.080 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.076+0000 7fe51bfff640 1 -- 172.21.15.72:0/2554278530 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3415557447 0 0) 0x7fe520004eb0 con 0x7fe5341457f0 2024-03-29T11:36:46.080 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.076+0000 7fe51bfff640 1 -- 172.21.15.72:0/2554278530 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe534142df0 con 0x7fe5341457f0 2024-03-29T11:36:46.080 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.076+0000 7fe51bfff640 1 -- 172.21.15.72:0/2554278530 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fe52c001b70 con 0x7fe5341457f0 2024-03-29T11:36:46.080 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.076+0000 7fe51bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:46.080 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.076+0000 7fe51bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:46.080 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.076+0000 7fe51bfff640 10 monclient(hunting): dump: 2024-03-29T11:36:46.080 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:36:46.080 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:36:46.080 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:46.080 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:46.080 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:46.080 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:46.080 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:36:46.080 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:36:46.080 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:36:46.081 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:46.081 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.076+0000 7fe51bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:46.081 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.076+0000 7fe51bfff640 1 -- 172.21.15.72:0/2554278530 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1082163309 0 0) 0x7fe534142ba0 con 0x7fe53413f650 2024-03-29T11:36:46.081 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.077+0000 7fe51bfff640 1 -- 172.21.15.72:0/2554278530 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe520004eb0 con 0x7fe53413f650 2024-03-29T11:36:46.081 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.077+0000 7fe51bfff640 1 -- 172.21.15.72:0/2554278530 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fe5280019b0 con 0x7fe53413f650 2024-03-29T11:36:46.081 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.077+0000 7fe51bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:46.081 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.077+0000 7fe51bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:46.081 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.077+0000 7fe51bfff640 10 monclient(hunting): dump: 2024-03-29T11:36:46.081 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:36:46.081 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:36:46.081 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:46.081 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:46.081 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:46.081 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:46.081 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:36:46.082 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:36:46.082 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:36:46.082 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:46.082 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.077+0000 7fe51bfff640 1 -- 172.21.15.72:0/2554278530 <== mon.0 v1:172.21.15.72:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 326798648 0 0) 0x7fe534142df0 con 0x7fe5341457f0 2024-03-29T11:36:46.082 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.077+0000 7fe51bfff640 1 -- 172.21.15.72:0/2554278530 >> v1:172.21.15.72:6790/0 conn(0x7fe5340a9d70 legacy=0x7fe534140a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:46.082 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.077+0000 7fe51bfff640 1 -- 172.21.15.72:0/2554278530 >> v1:172.21.15.89:6789/0 conn(0x7fe53413f650 legacy=0x7fe534144870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:46.082 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.077+0000 7fe51bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:36:46.082 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.077+0000 7fe51bfff640 1 monclient: found mon.a 2024-03-29T11:36:46.082 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.077+0000 7fe51bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:46.082 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.077+0000 7fe51bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:36:46.082 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.077+0000 7fe51bfff640 1 -- 172.21.15.72:0/2554278530 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe534071cb0 con 0x7fe5341457f0 2024-03-29T11:36:46.082 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.077+0000 7fe51bfff640 10 monclient: _finish_auth 0 2024-03-29T11:36:46.082 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.077+0000 7fe51bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:36:46.082 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.077+0000 7fe51bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:46.082 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.077+0000 7fe53a225640 5 monclient: authenticate success, global_id 4284 2024-03-29T11:36:46.083 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.077+0000 7fe53a225640 10 monclient: _renew_subs 2024-03-29T11:36:46.083 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.077+0000 7fe53a225640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:36:46.083 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.077+0000 7fe53a225640 1 -- 172.21.15.72:0/2554278530 --> v1:172.21.15.72:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe53413fda0 con 0x7fe5341457f0 2024-03-29T11:36:46.083 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.077+0000 7fe53a225640 10 client.4284.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:46.083 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.077+0000 7fe53a225640 10 monclient: _renew_subs 2024-03-29T11:36:46.083 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.077+0000 7fe53a225640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:36:46.083 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.077+0000 7fe53a225640 1 -- 172.21.15.72:0/2554278530 --> v1:172.21.15.72:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5340b1860 con 0x7fe5341457f0 2024-03-29T11:36:46.083 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.077+0000 7fe53a225640 1 librados: init done 2024-03-29T11:36:46.083 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.078+0000 7fe51bfff640 1 -- 172.21.15.72:0/2554278530 <== mon.0 v1:172.21.15.72:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe52c002080 con 0x7fe5341457f0 2024-03-29T11:36:46.083 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.078+0000 7fe51bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:46.083 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.078+0000 7fe51bfff640 1 -- 172.21.15.72:0/2554278530 <== mon.0 v1:172.21.15.72:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fe52c003bb0 con 0x7fe5341457f0 2024-03-29T11:36:46.083 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.078+0000 7fe51bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:46.083 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.078+0000 7fe51bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:46.083 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.078+0000 7fe51bfff640 10 monclient: dump: 2024-03-29T11:36:46.083 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:36:46.084 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:36:46.084 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:46.084 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:46.084 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:46.084 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:46.084 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:36:46.084 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:36:46.084 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:36:46.084 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:46.084 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.078+0000 7fe531ffb640 4 set_mon_vals no callback set 2024-03-29T11:36:46.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.081+0000 7fe53a225640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:46.086 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.081+0000 7fe53a225640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:46.086 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.081+0000 7fe53a225640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:36:46.086 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.081+0000 7fe53a225640 1 -- 172.21.15.72:0/2554278530 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe534075f10 con 0x7fe5341457f0 2024-03-29T11:36:46.086 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.082+0000 7fe51bfff640 1 -- 172.21.15.72:0/2554278530 <== mon.0 v1:172.21.15.72:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7fe52c019b80 con 0x7fe5341457f0 2024-03-29T11:36:46.086 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.083+0000 7fe51bfff640 1 -- 172.21.15.72:0/2554278530 <== mon.0 v1:172.21.15.72:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7fe52c083a50 con 0x7fe5341457f0 2024-03-29T11:36:46.086 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.083+0000 7fe5337fe640 10 monclient: get_auth_request con 0x7fe52006df20 auth_method 0 2024-03-29T11:36:46.086 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.083+0000 7fe51bfff640 10 client.4284.objecter ms_dispatch 0x7fe534000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:46.086 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.084+0000 7fe51bfff640 3 client.4284.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:46.086 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.084+0000 7fe51bfff640 3 client.4284.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:46.086 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.084+0000 7fe51bfff640 20 client.4284.objecter dump_active .. 0 homeless 2024-03-29T11:36:46.086 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.084+0000 7fe51bfff640 10 client.4284.objecter ms_handle_connect 0x7fe52006df20 2024-03-29T11:36:46.090 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.091+0000 7fe51bfff640 1 -- 172.21.15.72:0/2554278530 <== mon.0 v1:172.21.15.72:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe52c04f330 con 0x7fe5341457f0 2024-03-29T11:36:46.091 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.091+0000 7fe51bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:46.091 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.091+0000 7fe51bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:46.414 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.414+0000 7fe53a225640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:46.414 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.414+0000 7fe53a225640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:46.414 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.414+0000 7fe53a225640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:36:46.414 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.414+0000 7fe53a225640 1 -- 172.21.15.72:0/2554278530 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe534071e40 con 0x7fe5341457f0 2024-03-29T11:36:46.416 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.416+0000 7fe51bfff640 1 -- 172.21.15.72:0/2554278530 <== mon.0 v1:172.21.15.72:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7fe52c051260 con 0x7fe5341457f0 2024-03-29T11:36:46.416 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.416+0000 7fe51bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:46.416 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.416+0000 7fe51bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:46.417 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:36:46.418 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:46.422 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.422+0000 7fe53a225640 10 librados: watch_flush enter 2024-03-29T11:36:46.422 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.422+0000 7fe53a225640 10 librados: watch_flush exit 2024-03-29T11:36:46.422 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.422+0000 7fe53a225640 20 client.4284.objecter shutdown clearing up homeless session... 2024-03-29T11:36:46.422 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.422+0000 7fe53a225640 10 client.4284.objecter successfully canceled tick 2024-03-29T11:36:46.422 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.422+0000 7fe53a225640 1 -- 172.21.15.72:0/2554278530 >> v1:172.21.15.72:6816/1578866102 conn(0x7fe52006df20 legacy=0x7fe5200703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:46.422 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.422+0000 7fe53a225640 10 monclient: shutdown 2024-03-29T11:36:46.422 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.422+0000 7fe53a225640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:46.422 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.422+0000 7fe53a225640 1 -- 172.21.15.72:0/2554278530 >> v1:172.21.15.72:6789/0 conn(0x7fe5341457f0 legacy=0x7fe5341eec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:46.422 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.422+0000 7fe53a225640 1 -- 172.21.15.72:0/2554278530 shutdown_connections 2024-03-29T11:36:46.422 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.422+0000 7fe53a225640 1 -- 172.21.15.72:0/2554278530 >> 172.21.15.72:0/2554278530 conn(0x7fe534066a00 msgr2=0x7fe5341da870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:46.422 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.422+0000 7fe53a225640 1 -- 172.21.15.72:0/2554278530 shutdown_connections 2024-03-29T11:36:46.423 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.422+0000 7fe53a225640 1 -- 172.21.15.72:0/2554278530 wait complete. 2024-03-29T11:36:46.423 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:46.423+0000 7fe53a225640 1 librados: shutdown 2024-03-29T11:36:46.439 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:53.246 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:53.403 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.402+0000 7fae7ec38640 10 monclient: get_monmap_and_config 2024-03-29T11:36:53.403 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.402+0000 7fae7ec38640 10 monclient: build_initial_monmap 2024-03-29T11:36:53.403 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.402+0000 7fae7ec38640 10 monclient: monmap: 2024-03-29T11:36:53.403 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:36:53.403 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:36:53.403 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:36:53.403593+0000 2024-03-29T11:36:53.403 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:36:53.403593+0000 2024-03-29T11:36:53.404 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:53.404 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:53.404 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:36:53.404 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:36:53.404 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:36:53.404 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:53.408 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.404+0000 7fae7ec38640 1 Processor -- start 2024-03-29T11:36:53.408 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.404+0000 7fae7ec38640 1 -- start start 2024-03-29T11:36:53.408 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.404+0000 7fae7ec38640 10 monclient: init 2024-03-29T11:36:53.409 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.405+0000 7fae7ec38640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:53.409 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.405+0000 7fae7ec38640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:36:53.409 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.405+0000 7fae7ec38640 10 monclient(hunting): picked mon.noname-c con 0x7fae780aae30 addr v1:172.21.15.89:6789/0 2024-03-29T11:36:53.409 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.405+0000 7fae7ec38640 10 monclient(hunting): picked mon.noname-b con 0x7fae780a9e70 addr v1:172.21.15.72:6790/0 2024-03-29T11:36:53.409 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.408+0000 7fae7ec38640 10 monclient(hunting): picked mon.noname-a con 0x7fae78146290 addr v1:172.21.15.72:6789/0 2024-03-29T11:36:53.409 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.408+0000 7fae7ec38640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae780ac580 con 0x7fae78146290 2024-03-29T11:36:53.409 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.408+0000 7fae7ec38640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae780ac9a0 con 0x7fae780aae30 2024-03-29T11:36:53.409 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.408+0000 7fae7ec38640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae780ad640 con 0x7fae780a9e70 2024-03-29T11:36:53.409 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.408+0000 7fae7ec38640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:53.409 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.408+0000 7fae7ec38640 10 monclient(hunting): authenticate will time out at 972.696378s 2024-03-29T11:36:53.410 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.410+0000 7fae7d1d5640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fae78146290 0x7fae78151ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37791/0 (socket says 172.21.15.72:37791) 2024-03-29T11:36:53.410 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.410+0000 7fae7d1d5640 1 -- 172.21.15.72:0/2270047135 learned_addr learned my addr 172.21.15.72:0/2270047135 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:36:53.411 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.412+0000 7fae777fe640 1 -- 172.21.15.72:0/2270047135 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 385031649 0 0) 0x7fae780ac580 con 0x7fae78146290 2024-03-29T11:36:53.412 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.412+0000 7fae777fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:53.412 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.412+0000 7fae777fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:53.412 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.412+0000 7fae777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:53.412 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.412+0000 7fae777fe640 10 monclient(hunting): my global_id is 4287 2024-03-29T11:36:53.412 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.412+0000 7fae777fe640 1 -- 172.21.15.72:0/2270047135 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae68003520 con 0x7fae78146290 2024-03-29T11:36:53.414 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.413+0000 7fae777fe640 1 -- 172.21.15.72:0/2270047135 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1298143317 0 0) 0x7fae68003520 con 0x7fae78146290 2024-03-29T11:36:53.414 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.413+0000 7fae777fe640 1 -- 172.21.15.72:0/2270047135 >> v1:172.21.15.72:6790/0 conn(0x7fae780a9e70 legacy=0x7fae780acf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:53.414 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.413+0000 7fae777fe640 1 -- 172.21.15.72:0/2270047135 >> v1:172.21.15.89:6789/0 conn(0x7fae780aae30 legacy=0x7fae78146aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:53.414 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.413+0000 7fae777fe640 10 monclient: _finish_hunting 0 2024-03-29T11:36:53.414 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.413+0000 7fae777fe640 1 monclient: found mon.noname-a 2024-03-29T11:36:53.414 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.413+0000 7fae777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:53.415 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.413+0000 7fae777fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.72:6789/0 2024-03-29T11:36:53.415 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.413+0000 7fae777fe640 1 -- 172.21.15.72:0/2270047135 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae780ade20 con 0x7fae78146290 2024-03-29T11:36:53.415 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.413+0000 7fae777fe640 10 monclient: _finish_auth 0 2024-03-29T11:36:53.415 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.413+0000 7fae777fe640 10 monclient: _check_auth_tickets 2024-03-29T11:36:53.415 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.413+0000 7fae777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:53.415 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.413+0000 7fae7ec38640 5 monclient: authenticate success, global_id 4287 2024-03-29T11:36:53.415 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.413+0000 7fae7ec38640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:53.415 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.413+0000 7fae777fe640 1 -- 172.21.15.72:0/2270047135 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fae70001a40 con 0x7fae78146290 2024-03-29T11:36:53.415 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.413+0000 7fae777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:53.415 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.413+0000 7fae777fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:53.416 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.413+0000 7fae777fe640 10 monclient: dump: 2024-03-29T11:36:53.416 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:36:53.416 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:36:53.416 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:53.416 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:53.416 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:53.416 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:53.416 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:36:53.416 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:36:53.416 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:36:53.416 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:53.416 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.413+0000 7fae7ec38640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:53.416 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.413+0000 7fae777fe640 1 -- 172.21.15.72:0/2270047135 <== mon.0 v1:172.21.15.72:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae70001cd0 con 0x7fae78146290 2024-03-29T11:36:53.416 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.413+0000 7fae777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:53.416 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.413+0000 7fae7ec38640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:53.416 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.413+0000 7fae7ec38640 4 set_mon_vals no callback set 2024-03-29T11:36:53.417 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.414+0000 7fae777fe640 1 -- 172.21.15.72:0/2270047135 <== mon.0 v1:172.21.15.72:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fae70002060 con 0x7fae78146290 2024-03-29T11:36:53.417 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.414+0000 7fae777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:53.417 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.414+0000 7fae777fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:53.417 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.414+0000 7fae777fe640 10 monclient: dump: 2024-03-29T11:36:53.417 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:36:53.417 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:36:53.417 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:53.417 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:53.417 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:53.417 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:53.417 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:36:53.417 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:36:53.417 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:36:53.417 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:53.417 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.414+0000 7fae7ec38640 10 monclient: shutdown 2024-03-29T11:36:53.417 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.414+0000 7fae7ec38640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:53.418 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.414+0000 7fae7ec38640 1 -- 172.21.15.72:0/2270047135 >> v1:172.21.15.72:6789/0 conn(0x7fae78146290 legacy=0x7fae78151ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:53.418 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.414+0000 7fae7ec38640 1 -- 172.21.15.72:0/2270047135 shutdown_connections 2024-03-29T11:36:53.418 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.414+0000 7fae7ec38640 1 -- 172.21.15.72:0/2270047135 >> 172.21.15.72:0/2270047135 conn(0x7fae780b3c80 msgr2=0x7fae780b4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:53.418 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.414+0000 7fae7ec38640 1 -- 172.21.15.72:0/2270047135 shutdown_connections 2024-03-29T11:36:53.418 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.414+0000 7fae7ec38640 1 -- 172.21.15.72:0/2270047135 wait complete. 2024-03-29T11:36:53.419 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.415+0000 7fae7ec38640 10 monclient: build_initial_monmap 2024-03-29T11:36:53.419 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.415+0000 7fae7ec38640 10 monclient: monmap: 2024-03-29T11:36:53.419 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:36:53.419 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:53.420 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:36:53.420 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:36:53.420 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:53.420 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:53.420 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:36:53.420 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:36:53.420 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:36:53.420 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:53.420 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.415+0000 7fae7ec38640 1 librados: starting msgr at 2024-03-29T11:36:53.420 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.415+0000 7fae7ec38640 1 librados: starting objecter 2024-03-29T11:36:53.420 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.416+0000 7fae7ec38640 1 Processor -- start 2024-03-29T11:36:53.420 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.416+0000 7fae7ec38640 1 -- start start 2024-03-29T11:36:53.420 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.416+0000 7fae7ec38640 1 librados: setting wanted keys 2024-03-29T11:36:53.420 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.416+0000 7fae7ec38640 1 librados: calling monclient init 2024-03-29T11:36:53.420 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.416+0000 7fae7ec38640 10 monclient: init 2024-03-29T11:36:53.420 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.417+0000 7fae7ec38640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:53.421 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.417+0000 7fae7ec38640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:36:53.421 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.417+0000 7fae7ec38640 10 monclient(hunting): picked mon.noname-a con 0x7fae780aae30 addr v1:172.21.15.72:6789/0 2024-03-29T11:36:53.421 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.420+0000 7fae7ec38640 10 monclient(hunting): picked mon.noname-b con 0x7fae78146290 addr v1:172.21.15.89:6789/0 2024-03-29T11:36:53.421 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.422+0000 7fae7ec38640 10 monclient(hunting): picked mon.noname-c con 0x7fae78148910 addr v1:172.21.15.72:6790/0 2024-03-29T11:36:53.421 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.422+0000 7fae7ec38640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae7813f0a0 con 0x7fae780aae30 2024-03-29T11:36:53.421 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.422+0000 7fae7ec38640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae7813f380 con 0x7fae78146290 2024-03-29T11:36:53.421 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.422+0000 7fae7ec38640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae781494e0 con 0x7fae78148910 2024-03-29T11:36:53.421 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.422+0000 7fae7ec38640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:53.421 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.422+0000 7fae7ec38640 10 monclient(hunting): authenticate will time out at 972.710288s 2024-03-29T11:36:53.427 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.424+0000 7fae77fff640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fae78146290 0x7fae7813fa80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:37105/0 (socket says 172.21.15.72:37105) 2024-03-29T11:36:53.427 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.424+0000 7fae77fff640 1 -- 172.21.15.72:0/1085568566 learned_addr learned my addr 172.21.15.72:0/1085568566 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:36:53.427 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.424+0000 7fae757fa640 10 client.?.objecter ms_handle_connect 0x7fae78148910 2024-03-29T11:36:53.427 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.424+0000 7fae757fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:53.427 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.424+0000 7fae757fa640 1 -- 172.21.15.72:0/1085568566 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1422233240 0 0) 0x7fae781494e0 con 0x7fae78148910 2024-03-29T11:36:53.427 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.424+0000 7fae757fa640 10 monclient(hunting): _negotiate 2024-03-29T11:36:53.427 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.424+0000 7fae757fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:53.427 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.424+0000 7fae757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:53.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.424+0000 7fae757fa640 10 monclient(hunting): my global_id is 4295 2024-03-29T11:36:53.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.424+0000 7fae757fa640 1 -- 172.21.15.72:0/1085568566 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae5c004eb0 con 0x7fae78148910 2024-03-29T11:36:53.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.424+0000 7fae757fa640 10 client.?.objecter ms_handle_connect 0x7fae78146290 2024-03-29T11:36:53.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.424+0000 7fae757fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:53.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.424+0000 7fae757fa640 1 -- 172.21.15.72:0/1085568566 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3585763529 0 0) 0x7fae5c004eb0 con 0x7fae78148910 2024-03-29T11:36:53.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.424+0000 7fae757fa640 1 -- 172.21.15.72:0/1085568566 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae781494e0 con 0x7fae78148910 2024-03-29T11:36:53.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.424+0000 7fae757fa640 1 -- 172.21.15.72:0/1085568566 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fae70001900 con 0x7fae78148910 2024-03-29T11:36:53.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.424+0000 7fae757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:53.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.424+0000 7fae757fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:53.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.424+0000 7fae757fa640 10 monclient(hunting): dump: 2024-03-29T11:36:53.428 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:36:53.428 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:36:53.428 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:53.428 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:53.428 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:53.429 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:53.429 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:36:53.429 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:36:53.429 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:36:53.429 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:53.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.424+0000 7fae757fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:53.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.425+0000 7fae757fa640 1 -- 172.21.15.72:0/1085568566 <== mon.2 v1:172.21.15.72:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 407617412 0 0) 0x7fae781494e0 con 0x7fae78148910 2024-03-29T11:36:53.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.425+0000 7fae757fa640 1 -- 172.21.15.72:0/1085568566 >> v1:172.21.15.89:6789/0 conn(0x7fae78146290 legacy=0x7fae7813fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:53.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.425+0000 7fae757fa640 1 -- 172.21.15.72:0/1085568566 >> v1:172.21.15.72:6789/0 conn(0x7fae780aae30 legacy=0x7fae78149a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:53.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.425+0000 7fae757fa640 10 monclient: _finish_hunting 0 2024-03-29T11:36:53.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.425+0000 7fae757fa640 1 monclient: found mon.c 2024-03-29T11:36:53.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.425+0000 7fae757fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:53.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.425+0000 7fae757fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:36:53.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.425+0000 7fae757fa640 1 -- 172.21.15.72:0/1085568566 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae78140190 con 0x7fae78148910 2024-03-29T11:36:53.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.425+0000 7fae757fa640 10 monclient: _finish_auth 0 2024-03-29T11:36:53.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.425+0000 7fae757fa640 10 monclient: _check_auth_tickets 2024-03-29T11:36:53.430 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.425+0000 7fae757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:53.430 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.425+0000 7fae757fa640 1 -- 172.21.15.72:0/1085568566 <== mon.2 v1:172.21.15.72:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae70004e40 con 0x7fae78148910 2024-03-29T11:36:53.430 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.425+0000 7fae757fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:53.430 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.425+0000 7fae757fa640 1 -- 172.21.15.72:0/1085568566 <== mon.2 v1:172.21.15.72:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fae70005150 con 0x7fae78148910 2024-03-29T11:36:53.430 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.425+0000 7fae757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:53.430 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.425+0000 7fae757fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:53.430 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.425+0000 7fae757fa640 10 monclient: dump: 2024-03-29T11:36:53.430 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:36:53.430 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:36:53.430 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:53.430 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:36:53.430 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:53.430 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:36:53.430 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:36:53.430 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:36:53.431 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:36:53.431 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:36:53.431 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.426+0000 7fae7ec38640 5 monclient: authenticate success, global_id 4295 2024-03-29T11:36:53.431 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.426+0000 7fae7ec38640 10 monclient: _renew_subs 2024-03-29T11:36:53.431 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.426+0000 7fae7ec38640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:36:53.431 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.426+0000 7fae7ec38640 1 -- 172.21.15.72:0/1085568566 --> v1:172.21.15.72:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae78139d20 con 0x7fae78148910 2024-03-29T11:36:53.431 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.426+0000 7fae7ec38640 10 client.4295.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:53.431 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.426+0000 7fae7ec38640 10 monclient: _renew_subs 2024-03-29T11:36:53.431 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.426+0000 7fae7ec38640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:36:53.431 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.426+0000 7fae7ec38640 1 -- 172.21.15.72:0/1085568566 --> v1:172.21.15.72:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fae78089a70 con 0x7fae78148910 2024-03-29T11:36:53.431 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.426+0000 7fae7ec38640 1 librados: init done 2024-03-29T11:36:53.431 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.428+0000 7fae76ffd640 4 set_mon_vals no callback set 2024-03-29T11:36:53.431 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.428+0000 7fae62ffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:53.432 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.430+0000 7fae757fa640 1 -- 172.21.15.72:0/1085568566 <== mon.2 v1:172.21.15.72:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7fae7001b120 con 0x7fae78148910 2024-03-29T11:36:53.432 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.430+0000 7fae757fa640 1 -- 172.21.15.72:0/1085568566 <== mon.2 v1:172.21.15.72:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7fae70084e20 con 0x7fae78148910 2024-03-29T11:36:53.432 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.430+0000 7fae757fa640 10 client.4295.objecter ms_dispatch 0x7fae78000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:53.433 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.430+0000 7fae757fa640 3 client.4295.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:53.433 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.430+0000 7fae757fa640 3 client.4295.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:53.433 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.430+0000 7fae757fa640 20 client.4295.objecter dump_active .. 0 homeless 2024-03-29T11:36:53.433 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.432+0000 7fae62ffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:53.433 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.432+0000 7fae62ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:36:53.433 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.432+0000 7fae62ffd640 1 -- 172.21.15.72:0/1085568566 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae780008d0 con 0x7fae78148910 2024-03-29T11:36:53.433 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.432+0000 7fae7c9d4640 10 monclient: get_auth_request con 0x7fae5c06dcf0 auth_method 0 2024-03-29T11:36:53.433 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.433+0000 7fae757fa640 10 client.4295.objecter ms_handle_connect 0x7fae5c06dcf0 2024-03-29T11:36:53.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.440+0000 7fae757fa640 1 -- 172.21.15.72:0/1085568566 <== mon.2 v1:172.21.15.72:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fae70050700 con 0x7fae78148910 2024-03-29T11:36:53.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.440+0000 7fae757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:53.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.440+0000 7fae757fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:53.761 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.760+0000 7fae62ffd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:53.761 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.760+0000 7fae62ffd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:53.761 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.760+0000 7fae62ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:36:53.761 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.760+0000 7fae62ffd640 1 -- 172.21.15.72:0/1085568566 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae780b2880 con 0x7fae78148910 2024-03-29T11:36:53.763 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:36:53.764 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:53.764 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.764+0000 7fae757fa640 1 -- 172.21.15.72:0/1085568566 <== mon.2 v1:172.21.15.72:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7fae70052630 con 0x7fae78148910 2024-03-29T11:36:53.764 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.764+0000 7fae757fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:53.764 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.764+0000 7fae757fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:53.767 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.767+0000 7fae62ffd640 10 librados: watch_flush enter 2024-03-29T11:36:53.767 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.767+0000 7fae62ffd640 10 librados: watch_flush exit 2024-03-29T11:36:53.768 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.767+0000 7fae62ffd640 20 client.4295.objecter shutdown clearing up homeless session... 2024-03-29T11:36:53.768 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.767+0000 7fae62ffd640 10 client.4295.objecter successfully canceled tick 2024-03-29T11:36:53.768 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.767+0000 7fae62ffd640 1 -- 172.21.15.72:0/1085568566 >> v1:172.21.15.72:6816/1578866102 conn(0x7fae5c06dcf0 legacy=0x7fae5c0701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:53.768 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.767+0000 7fae62ffd640 10 monclient: shutdown 2024-03-29T11:36:53.768 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.767+0000 7fae62ffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:53.768 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.767+0000 7fae62ffd640 1 -- 172.21.15.72:0/1085568566 >> v1:172.21.15.72:6790/0 conn(0x7fae78148910 legacy=0x7fae78088350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:53.768 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.768+0000 7fae62ffd640 1 -- 172.21.15.72:0/1085568566 shutdown_connections 2024-03-29T11:36:53.768 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.768+0000 7fae62ffd640 1 -- 172.21.15.72:0/1085568566 >> 172.21.15.72:0/1085568566 conn(0x7fae780b3c80 msgr2=0x7fae780b2da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:53.769 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.769+0000 7fae62ffd640 1 -- 172.21.15.72:0/1085568566 shutdown_connections 2024-03-29T11:36:53.771 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.772+0000 7fae62ffd640 1 -- 172.21.15.72:0/1085568566 wait complete. 2024-03-29T11:36:53.771 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:36:53.772+0000 7fae62ffd640 1 librados: shutdown 2024-03-29T11:36:53.788 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:00.595 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:00.747 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.747+0000 7f62d299a640 10 monclient: get_monmap_and_config 2024-03-29T11:37:00.748 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.747+0000 7f62d299a640 10 monclient: build_initial_monmap 2024-03-29T11:37:00.748 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.747+0000 7f62d299a640 10 monclient: monmap: 2024-03-29T11:37:00.748 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:37:00.748 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:00.748 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:37:00.748370+0000 2024-03-29T11:37:00.748 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:37:00.748370+0000 2024-03-29T11:37:00.748 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:00.748 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:00.748 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:37:00.748 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:37:00.749 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:37:00.749 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:00.751 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.750+0000 7f62d299a640 1 Processor -- start 2024-03-29T11:37:00.751 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.752+0000 7f62d299a640 1 -- start start 2024-03-29T11:37:00.752 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.752+0000 7f62d299a640 10 monclient: init 2024-03-29T11:37:00.755 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.753+0000 7f62d299a640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:00.755 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.753+0000 7f62d299a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:37:00.755 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.753+0000 7f62d299a640 10 monclient(hunting): picked mon.noname-c con 0x7f62cc0aa2a0 addr v1:172.21.15.89:6789/0 2024-03-29T11:37:00.757 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.757+0000 7f62d299a640 10 monclient(hunting): picked mon.noname-a con 0x7f62cc141f20 addr v1:172.21.15.72:6789/0 2024-03-29T11:37:00.757 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.757+0000 7f62cb7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f62cc141f20 0x7f62cc142340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:54795/0 (socket says 172.21.15.72:54795) 2024-03-29T11:37:00.757 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.757+0000 7f62cb7fe640 1 -- 172.21.15.72:0/815578730 learned_addr learned my addr 172.21.15.72:0/815578730 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:37:00.757 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.757+0000 7f62d299a640 10 monclient(hunting): picked mon.noname-b con 0x7f62cc13b3c0 addr v1:172.21.15.72:6790/0 2024-03-29T11:37:00.757 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.757+0000 7f62d299a640 1 -- 172.21.15.72:0/815578730 --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62cc13e5c0 con 0x7f62cc141f20 2024-03-29T11:37:00.757 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.757+0000 7f62d299a640 1 -- 172.21.15.72:0/815578730 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62cc0ab190 con 0x7f62cc0aa2a0 2024-03-29T11:37:00.757 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.757+0000 7f62d299a640 1 -- 172.21.15.72:0/815578730 --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62cc0ab470 con 0x7f62cc13b3c0 2024-03-29T11:37:00.757 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.757+0000 7f62d299a640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:00.757 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.757+0000 7f62d299a640 10 monclient(hunting): authenticate will time out at 980.045827s 2024-03-29T11:37:00.761 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.762+0000 7f62caffd640 1 -- 172.21.15.72:0/815578730 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2133101274 0 0) 0x7f62cc0ab190 con 0x7f62cc0aa2a0 2024-03-29T11:37:00.761 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.762+0000 7f62caffd640 10 monclient(hunting): _negotiate 2024-03-29T11:37:00.761 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.762+0000 7f62caffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:00.761 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.762+0000 7f62caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:00.761 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.762+0000 7f62caffd640 10 monclient(hunting): my global_id is 4294 2024-03-29T11:37:00.761 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.762+0000 7f62caffd640 1 -- 172.21.15.72:0/815578730 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62b4003520 con 0x7f62cc0aa2a0 2024-03-29T11:37:00.762 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.763+0000 7f62caffd640 1 -- 172.21.15.72:0/815578730 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3513411417 0 0) 0x7f62cc13e5c0 con 0x7f62cc141f20 2024-03-29T11:37:00.762 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.763+0000 7f62caffd640 10 monclient(hunting): _negotiate 2024-03-29T11:37:00.762 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.763+0000 7f62caffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:00.762 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.763+0000 7f62caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:00.762 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.763+0000 7f62caffd640 10 monclient(hunting): my global_id is 4290 2024-03-29T11:37:00.762 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.763+0000 7f62caffd640 1 -- 172.21.15.72:0/815578730 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62b4003840 con 0x7f62cc141f20 2024-03-29T11:37:00.762 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.763+0000 7f62caffd640 1 -- 172.21.15.72:0/815578730 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4025853578 0 0) 0x7f62b4003520 con 0x7f62cc0aa2a0 2024-03-29T11:37:00.762 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.763+0000 7f62caffd640 1 -- 172.21.15.72:0/815578730 >> v1:172.21.15.72:6790/0 conn(0x7f62cc13b3c0 legacy=0x7f62cc13b810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:37:00.763 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.764+0000 7f62caffd640 1 -- 172.21.15.72:0/815578730 >> v1:172.21.15.72:6789/0 conn(0x7f62cc141f20 legacy=0x7f62cc142340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:00.764 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.764+0000 7f62caffd640 10 monclient: _finish_hunting 0 2024-03-29T11:37:00.764 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.764+0000 7f62caffd640 1 monclient: found mon.noname-c 2024-03-29T11:37:00.764 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.764+0000 7f62caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:00.764 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.764+0000 7f62caffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.89:6789/0 2024-03-29T11:37:00.764 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.764+0000 7f62caffd640 1 -- 172.21.15.72:0/815578730 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62cc142a50 con 0x7f62cc0aa2a0 2024-03-29T11:37:00.764 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.764+0000 7f62caffd640 10 monclient: _finish_auth 0 2024-03-29T11:37:00.764 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.764+0000 7f62caffd640 10 monclient: _check_auth_tickets 2024-03-29T11:37:00.764 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.764+0000 7f62caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:00.764 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.764+0000 7f62caffd640 1 -- 172.21.15.72:0/815578730 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f62c0001670 con 0x7f62cc0aa2a0 2024-03-29T11:37:00.764 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.764+0000 7f62caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:00.765 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.764+0000 7f62caffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:00.765 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.764+0000 7f62caffd640 10 monclient: dump: 2024-03-29T11:37:00.765 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:37:00.765 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:00.765 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:00.765 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:00.765 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:00.765 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:00.765 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:37:00.765 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:37:00.765 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:37:00.765 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:00.765 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.764+0000 7f62d299a640 5 monclient: authenticate success, global_id 4294 2024-03-29T11:37:00.765 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.764+0000 7f62d299a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:00.765 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.764+0000 7f62caffd640 1 -- 172.21.15.72:0/815578730 <== mon.1 v1:172.21.15.89:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62c0001850 con 0x7f62cc0aa2a0 2024-03-29T11:37:00.765 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.764+0000 7f62caffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:00.766 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.764+0000 7f62caffd640 1 -- 172.21.15.72:0/815578730 <== mon.1 v1:172.21.15.89:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f62c0001ca0 con 0x7f62cc0aa2a0 2024-03-29T11:37:00.766 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.764+0000 7f62caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:00.766 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.764+0000 7f62caffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:00.766 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.764+0000 7f62caffd640 10 monclient: dump: 2024-03-29T11:37:00.766 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:37:00.766 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:00.766 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:00.766 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:00.766 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:00.766 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:00.766 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:37:00.766 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:37:00.766 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:37:00.767 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:00.767 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.765+0000 7f62d299a640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:00.767 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.765+0000 7f62d299a640 4 set_mon_vals no callback set 2024-03-29T11:37:00.767 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.766+0000 7f62d299a640 10 monclient: shutdown 2024-03-29T11:37:00.767 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.766+0000 7f62d299a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:00.767 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.766+0000 7f62d299a640 1 -- 172.21.15.72:0/815578730 >> v1:172.21.15.89:6789/0 conn(0x7f62cc0aa2a0 legacy=0x7f62cc0ad270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:00.768 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.767+0000 7f62d299a640 1 -- 172.21.15.72:0/815578730 shutdown_connections 2024-03-29T11:37:00.768 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.767+0000 7f62d299a640 1 -- 172.21.15.72:0/815578730 >> 172.21.15.72:0/815578730 conn(0x7f62cc0b35e0 msgr2=0x7f62cc0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:00.768 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.768+0000 7f62d299a640 1 -- 172.21.15.72:0/815578730 shutdown_connections 2024-03-29T11:37:00.768 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.768+0000 7f62d299a640 1 -- 172.21.15.72:0/815578730 wait complete. 2024-03-29T11:37:00.769 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.769+0000 7f62d299a640 10 monclient: build_initial_monmap 2024-03-29T11:37:00.769 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.769+0000 7f62d299a640 10 monclient: monmap: 2024-03-29T11:37:00.770 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:37:00.770 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:00.770 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:37:00.770 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:37:00.770 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:00.770 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:00.770 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:37:00.770 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:37:00.770 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:37:00.770 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:00.770 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.770+0000 7f62d299a640 1 librados: starting msgr at 2024-03-29T11:37:00.770 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.770+0000 7f62d299a640 1 librados: starting objecter 2024-03-29T11:37:00.770 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.770+0000 7f62d299a640 1 Processor -- start 2024-03-29T11:37:00.770 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.770+0000 7f62d299a640 1 -- start start 2024-03-29T11:37:00.771 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.770+0000 7f62d299a640 1 librados: setting wanted keys 2024-03-29T11:37:00.771 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.770+0000 7f62d299a640 1 librados: calling monclient init 2024-03-29T11:37:00.771 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.770+0000 7f62d299a640 10 monclient: init 2024-03-29T11:37:00.777 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.771+0000 7f62d299a640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:00.777 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.771+0000 7f62d299a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:37:00.777 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.771+0000 7f62d299a640 10 monclient(hunting): picked mon.noname-a con 0x7f62cc0aa2a0 addr v1:172.21.15.72:6789/0 2024-03-29T11:37:00.777 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.773+0000 7f62d299a640 10 monclient(hunting): picked mon.noname-b con 0x7f62cc13b3c0 addr v1:172.21.15.89:6789/0 2024-03-29T11:37:00.777 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.773+0000 7f62d299a640 10 monclient(hunting): picked mon.noname-c con 0x7f62cc141f20 addr v1:172.21.15.72:6790/0 2024-03-29T11:37:00.777 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.773+0000 7f62d299a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62cc0707c0 con 0x7f62cc0aa2a0 2024-03-29T11:37:00.777 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.773+0000 7f62d299a640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62cc070aa0 con 0x7f62cc13b3c0 2024-03-29T11:37:00.777 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.773+0000 7f62d299a640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62cc06ead0 con 0x7f62cc141f20 2024-03-29T11:37:00.777 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.773+0000 7f62d299a640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:00.777 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.773+0000 7f62d299a640 10 monclient(hunting): authenticate will time out at 980.061908s 2024-03-29T11:37:00.777 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.777+0000 7f62cb7fe640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f62cc13b3c0 0x7f62cc0719b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:43401/0 (socket says 172.21.15.72:43401) 2024-03-29T11:37:00.777 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.777+0000 7f62cb7fe640 1 -- 172.21.15.72:0/1124066131 learned_addr learned my addr 172.21.15.72:0/1124066131 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:37:00.778 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.777+0000 7f62c8ff9640 10 client.?.objecter ms_handle_connect 0x7f62cc13b3c0 2024-03-29T11:37:00.778 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.777+0000 7f62c8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:00.778 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.777+0000 7f62c8ff9640 1 -- 172.21.15.72:0/1124066131 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 645842287 0 0) 0x7f62cc070aa0 con 0x7f62cc13b3c0 2024-03-29T11:37:00.778 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.777+0000 7f62c8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:37:00.778 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.777+0000 7f62c8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:00.778 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.777+0000 7f62c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:00.778 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.777+0000 7f62c8ff9640 10 monclient(hunting): my global_id is 4297 2024-03-29T11:37:00.778 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.777+0000 7f62c8ff9640 1 -- 172.21.15.72:0/1124066131 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62b8004eb0 con 0x7f62cc13b3c0 2024-03-29T11:37:00.778 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.777+0000 7f62c8ff9640 1 -- 172.21.15.72:0/1124066131 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4210508369 0 0) 0x7f62b8004eb0 con 0x7f62cc13b3c0 2024-03-29T11:37:00.780 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.778+0000 7f62c8ff9640 1 -- 172.21.15.72:0/1124066131 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62cc070aa0 con 0x7f62cc13b3c0 2024-03-29T11:37:00.780 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.778+0000 7f62c8ff9640 1 -- 172.21.15.72:0/1124066131 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f62c4001cb0 con 0x7f62cc13b3c0 2024-03-29T11:37:00.780 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.778+0000 7f62c8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:00.780 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.778+0000 7f62c8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:00.783 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.778+0000 7f62c8ff9640 10 monclient(hunting): dump: 2024-03-29T11:37:00.783 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:37:00.783 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:00.784 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:00.784 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:00.784 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:00.784 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:00.784 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:37:00.784 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:37:00.784 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:37:00.784 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:00.784 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.778+0000 7f62c8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:00.784 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.778+0000 7f62c8ff9640 1 -- 172.21.15.72:0/1124066131 <== mon.1 v1:172.21.15.89:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3089334381 0 0) 0x7f62cc070aa0 con 0x7f62cc13b3c0 2024-03-29T11:37:00.784 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.778+0000 7f62c8ff9640 1 -- 172.21.15.72:0/1124066131 >> v1:172.21.15.72:6790/0 conn(0x7f62cc141f20 legacy=0x7f62cc1ee9e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:37:00.784 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.780+0000 7f62c8ff9640 1 -- 172.21.15.72:0/1124066131 >> v1:172.21.15.72:6789/0 conn(0x7f62cc0aa2a0 legacy=0x7f62cc0712a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:00.784 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.780+0000 7f62c8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:37:00.784 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.780+0000 7f62c8ff9640 1 monclient: found mon.b 2024-03-29T11:37:00.784 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.780+0000 7f62c8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:00.784 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.780+0000 7f62c8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:37:00.785 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.780+0000 7f62c8ff9640 1 -- 172.21.15.72:0/1124066131 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62cc06e7a0 con 0x7f62cc13b3c0 2024-03-29T11:37:00.785 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.780+0000 7f62c8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:37:00.785 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.780+0000 7f62c8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:37:00.785 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.780+0000 7f62c8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:00.785 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.780+0000 7f62c8ff9640 10 client.?.objecter ms_handle_connect 0x7f62cc0aa2a0 2024-03-29T11:37:00.785 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.780+0000 7f62c8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:00.785 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.780+0000 7f62c8ff9640 1 -- 172.21.15.72:0/1124066131 <== mon.1 v1:172.21.15.89:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62c4001fd0 con 0x7f62cc13b3c0 2024-03-29T11:37:00.785 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.780+0000 7f62c8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:00.785 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.780+0000 7f62c8ff9640 1 -- 172.21.15.72:0/1124066131 <== mon.1 v1:172.21.15.89:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f62c4004fb0 con 0x7f62cc13b3c0 2024-03-29T11:37:00.785 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.780+0000 7f62c8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:00.785 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.780+0000 7f62c8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:00.785 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.780+0000 7f62c8ff9640 10 monclient: dump: 2024-03-29T11:37:00.785 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:37:00.785 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:00.785 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:00.785 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:00.786 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:00.786 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:00.786 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:37:00.786 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:37:00.786 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:37:00.786 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:00.786 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.780+0000 7f62c9ffb640 4 set_mon_vals no callback set 2024-03-29T11:37:00.786 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.781+0000 7f62d299a640 5 monclient: authenticate success, global_id 4297 2024-03-29T11:37:00.786 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.781+0000 7f62d299a640 10 monclient: _renew_subs 2024-03-29T11:37:00.786 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.781+0000 7f62d299a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:37:00.786 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.781+0000 7f62d299a640 1 -- 172.21.15.72:0/1124066131 --> v1:172.21.15.89:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62cc1418e0 con 0x7f62cc13b3c0 2024-03-29T11:37:00.786 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.781+0000 7f62d299a640 10 client.4297.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:00.786 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.781+0000 7f62d299a640 10 monclient: _renew_subs 2024-03-29T11:37:00.786 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.781+0000 7f62d299a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:37:00.786 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.781+0000 7f62d299a640 1 -- 172.21.15.72:0/1124066131 --> v1:172.21.15.89:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f62cc1d1940 con 0x7f62cc13b3c0 2024-03-29T11:37:00.787 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.781+0000 7f62d299a640 1 librados: init done 2024-03-29T11:37:00.788 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.783+0000 7f62b27fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:00.788 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.783+0000 7f62b27fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:00.788 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.783+0000 7f62b27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:37:00.788 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.783+0000 7f62b27fc640 1 -- 172.21.15.72:0/1124066131 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62cc075f10 con 0x7f62cc13b3c0 2024-03-29T11:37:00.788 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.786+0000 7f62c8ff9640 1 -- 172.21.15.72:0/1124066131 <== mon.1 v1:172.21.15.89:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f62c401af80 con 0x7f62cc13b3c0 2024-03-29T11:37:00.788 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.787+0000 7f62c8ff9640 1 -- 172.21.15.72:0/1124066131 <== mon.1 v1:172.21.15.89:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f62c4084ef0 con 0x7f62cc13b3c0 2024-03-29T11:37:00.788 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.787+0000 7f62c8ff9640 10 client.4297.objecter ms_dispatch 0x7f62cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:00.788 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.787+0000 7f62c8ff9640 3 client.4297.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:00.789 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.787+0000 7f62c8ff9640 3 client.4297.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:00.789 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.787+0000 7f62c8ff9640 20 client.4297.objecter dump_active .. 0 homeless 2024-03-29T11:37:00.789 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.787+0000 7f62cbfff640 10 monclient: get_auth_request con 0x7f62b806dbb0 auth_method 0 2024-03-29T11:37:00.789 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.788+0000 7f62c8ff9640 10 client.4297.objecter ms_handle_connect 0x7f62b806dbb0 2024-03-29T11:37:00.794 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.795+0000 7f62c8ff9640 1 -- 172.21.15.72:0/1124066131 <== mon.1 v1:172.21.15.89:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f62c40507d0 con 0x7f62cc13b3c0 2024-03-29T11:37:00.794 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.795+0000 7f62c8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:00.795 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:00.795+0000 7f62c8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:01.120 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:01.120+0000 7f62b27fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:01.120 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:01.120+0000 7f62b27fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:01.120 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:01.120+0000 7f62b27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:37:01.120 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:01.120+0000 7f62b27fc640 1 -- 172.21.15.72:0/1124066131 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62cc06ed80 con 0x7f62cc13b3c0 2024-03-29T11:37:01.121 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:01.122+0000 7f62c8ff9640 1 -- 172.21.15.72:0/1124066131 <== mon.1 v1:172.21.15.89:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f62c4052700 con 0x7f62cc13b3c0 2024-03-29T11:37:01.121 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:01.122+0000 7f62c8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:01.121 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:01.122+0000 7f62c8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:01.123 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:37:01.123 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:01.125 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:01.125+0000 7f62b27fc640 10 librados: watch_flush enter 2024-03-29T11:37:01.126 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:01.126+0000 7f62b27fc640 10 librados: watch_flush exit 2024-03-29T11:37:01.126 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:01.126+0000 7f62b27fc640 20 client.4297.objecter shutdown clearing up homeless session... 2024-03-29T11:37:01.126 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:01.126+0000 7f62b27fc640 10 client.4297.objecter successfully canceled tick 2024-03-29T11:37:01.126 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:01.126+0000 7f62b27fc640 1 -- 172.21.15.72:0/1124066131 >> v1:172.21.15.72:6816/1578866102 conn(0x7f62b806dbb0 legacy=0x7f62b8070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:01.126 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:01.126+0000 7f62b27fc640 10 monclient: shutdown 2024-03-29T11:37:01.126 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:01.126+0000 7f62b27fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:01.126 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:01.126+0000 7f62b27fc640 1 -- 172.21.15.72:0/1124066131 >> v1:172.21.15.89:6789/0 conn(0x7f62cc13b3c0 legacy=0x7f62cc0719b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:01.127 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:01.128+0000 7f62b27fc640 1 -- 172.21.15.72:0/1124066131 shutdown_connections 2024-03-29T11:37:01.127 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:01.128+0000 7f62b27fc640 1 -- 172.21.15.72:0/1124066131 >> 172.21.15.72:0/1124066131 conn(0x7f62cc0b35e0 msgr2=0x7f62cc0a97a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:01.128 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:01.129+0000 7f62b27fc640 1 -- 172.21.15.72:0/1124066131 shutdown_connections 2024-03-29T11:37:01.128 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:01.129+0000 7f62b27fc640 1 -- 172.21.15.72:0/1124066131 wait complete. 2024-03-29T11:37:01.128 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:01.129+0000 7f62b27fc640 1 librados: shutdown 2024-03-29T11:37:01.150 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:07.958 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:08.108 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.108+0000 7fe364e46640 10 monclient: get_monmap_and_config 2024-03-29T11:37:08.108 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.108+0000 7fe364e46640 10 monclient: build_initial_monmap 2024-03-29T11:37:08.109 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.109+0000 7fe364e46640 10 monclient: monmap: 2024-03-29T11:37:08.109 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:37:08.109 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:08.109 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:37:08.109938+0000 2024-03-29T11:37:08.109 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:37:08.109938+0000 2024-03-29T11:37:08.109 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:08.109 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:08.109 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:37:08.109 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:37:08.109 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:37:08.109 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:08.111 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.111+0000 7fe364e46640 1 Processor -- start 2024-03-29T11:37:08.111 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.112+0000 7fe364e46640 1 -- start start 2024-03-29T11:37:08.111 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.112+0000 7fe364e46640 10 monclient: init 2024-03-29T11:37:08.112 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.113+0000 7fe364e46640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:08.112 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.113+0000 7fe364e46640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:37:08.116 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.113+0000 7fe364e46640 10 monclient(hunting): picked mon.noname-c con 0x7fe358038c60 addr v1:172.21.15.89:6789/0 2024-03-29T11:37:08.116 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.113+0000 7fe364e46640 10 monclient(hunting): picked mon.noname-a con 0x7fe3580324b0 addr v1:172.21.15.72:6789/0 2024-03-29T11:37:08.116 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.113+0000 7fe364e46640 10 monclient(hunting): picked mon.noname-b con 0x7fe3580d11e0 addr v1:172.21.15.72:6790/0 2024-03-29T11:37:08.116 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.113+0000 7fe364e46640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3580caf80 con 0x7fe3580324b0 2024-03-29T11:37:08.116 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.113+0000 7fe364e46640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe358039850 con 0x7fe358038c60 2024-03-29T11:37:08.116 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.113+0000 7fe364e46640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3580c78d0 con 0x7fe3580d11e0 2024-03-29T11:37:08.116 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.113+0000 7fe364e46640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:08.116 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.113+0000 7fe364e46640 10 monclient(hunting): authenticate will time out at 987.401401s 2024-03-29T11:37:08.116 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.116+0000 7fe35f7fe640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe358038c60 0x7fe3580c4ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:48209/0 (socket says 172.21.15.72:48209) 2024-03-29T11:37:08.117 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.116+0000 7fe35f7fe640 1 -- 172.21.15.72:0/3280509347 learned_addr learned my addr 172.21.15.72:0/3280509347 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:37:08.119 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.118+0000 7fe35e7fc640 1 -- 172.21.15.72:0/3280509347 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2815327725 0 0) 0x7fe3580c78d0 con 0x7fe3580d11e0 2024-03-29T11:37:08.119 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.118+0000 7fe35e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:37:08.119 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.118+0000 7fe35e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:08.119 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.118+0000 7fe35e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:08.119 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.118+0000 7fe35e7fc640 10 monclient(hunting): my global_id is 4298 2024-03-29T11:37:08.119 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.118+0000 7fe35e7fc640 1 -- 172.21.15.72:0/3280509347 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe350003520 con 0x7fe3580d11e0 2024-03-29T11:37:08.119 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.118+0000 7fe35e7fc640 1 -- 172.21.15.72:0/3280509347 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2490785784 0 0) 0x7fe358039850 con 0x7fe358038c60 2024-03-29T11:37:08.119 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.118+0000 7fe35e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:37:08.121 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.118+0000 7fe35e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:08.122 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.118+0000 7fe35e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:08.122 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.118+0000 7fe35e7fc640 10 monclient(hunting): my global_id is 4300 2024-03-29T11:37:08.122 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.118+0000 7fe35e7fc640 1 -- 172.21.15.72:0/3280509347 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3580c78d0 con 0x7fe358038c60 2024-03-29T11:37:08.122 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.118+0000 7fe35e7fc640 1 -- 172.21.15.72:0/3280509347 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1527236491 0 0) 0x7fe3580caf80 con 0x7fe3580324b0 2024-03-29T11:37:08.122 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.118+0000 7fe35e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:37:08.122 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.118+0000 7fe35e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:08.122 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.118+0000 7fe35e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:08.122 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.118+0000 7fe35e7fc640 10 monclient(hunting): my global_id is 4296 2024-03-29T11:37:08.122 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.118+0000 7fe35e7fc640 1 -- 172.21.15.72:0/3280509347 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe358039850 con 0x7fe3580324b0 2024-03-29T11:37:08.122 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.118+0000 7fe35e7fc640 1 -- 172.21.15.72:0/3280509347 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 291828176 0 0) 0x7fe358039850 con 0x7fe3580324b0 2024-03-29T11:37:08.122 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.119+0000 7fe35e7fc640 1 -- 172.21.15.72:0/3280509347 >> v1:172.21.15.72:6790/0 conn(0x7fe3580d11e0 legacy=0x7fe3580d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:08.122 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.119+0000 7fe35e7fc640 1 -- 172.21.15.72:0/3280509347 >> v1:172.21.15.89:6789/0 conn(0x7fe358038c60 legacy=0x7fe3580c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:08.122 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.119+0000 7fe35e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:37:08.122 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.119+0000 7fe35e7fc640 1 monclient: found mon.noname-a 2024-03-29T11:37:08.123 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.119+0000 7fe35e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:08.123 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.119+0000 7fe35e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.72:6789/0 2024-03-29T11:37:08.123 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.119+0000 7fe35e7fc640 1 -- 172.21.15.72:0/3280509347 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3580c7a60 con 0x7fe3580324b0 2024-03-29T11:37:08.123 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.119+0000 7fe35e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:37:08.123 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.119+0000 7fe35e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:37:08.123 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.119+0000 7fe35e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:08.123 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.119+0000 7fe35e7fc640 1 -- 172.21.15.72:0/3280509347 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fe34c0014e0 con 0x7fe3580324b0 2024-03-29T11:37:08.123 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.119+0000 7fe35e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:08.123 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.119+0000 7fe35e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:08.123 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.119+0000 7fe35e7fc640 10 monclient: dump: 2024-03-29T11:37:08.123 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:37:08.123 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:08.123 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:08.123 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:08.123 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:08.123 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:08.124 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:37:08.124 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:37:08.124 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:37:08.124 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:08.124 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.119+0000 7fe35e7fc640 1 -- 172.21.15.72:0/3280509347 <== mon.0 v1:172.21.15.72:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe34c0016b0 con 0x7fe3580324b0 2024-03-29T11:37:08.124 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.119+0000 7fe35e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:08.124 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.119+0000 7fe35e7fc640 1 -- 172.21.15.72:0/3280509347 <== mon.0 v1:172.21.15.72:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fe34c001a40 con 0x7fe3580324b0 2024-03-29T11:37:08.124 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.119+0000 7fe35e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:08.124 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.119+0000 7fe35e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:08.124 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.119+0000 7fe35e7fc640 10 monclient: dump: 2024-03-29T11:37:08.124 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:37:08.124 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:08.124 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:08.124 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:08.124 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:08.124 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:08.125 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:37:08.125 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:37:08.125 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:37:08.125 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:08.125 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.120+0000 7fe364e46640 5 monclient: authenticate success, global_id 4296 2024-03-29T11:37:08.125 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.120+0000 7fe364e46640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:08.125 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.120+0000 7fe364e46640 4 set_mon_vals no callback set 2024-03-29T11:37:08.125 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.121+0000 7fe364e46640 10 monclient: shutdown 2024-03-29T11:37:08.125 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.121+0000 7fe364e46640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:08.125 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.121+0000 7fe364e46640 1 -- 172.21.15.72:0/3280509347 >> v1:172.21.15.72:6789/0 conn(0x7fe3580324b0 legacy=0x7fe3580cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:08.125 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.121+0000 7fe364e46640 1 -- 172.21.15.72:0/3280509347 shutdown_connections 2024-03-29T11:37:08.125 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.121+0000 7fe364e46640 1 -- 172.21.15.72:0/3280509347 >> 172.21.15.72:0/3280509347 conn(0x7fe358005c20 msgr2=0x7fe358006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:08.126 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.123+0000 7fe364e46640 1 -- 172.21.15.72:0/3280509347 shutdown_connections 2024-03-29T11:37:08.126 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.125+0000 7fe364e46640 1 -- 172.21.15.72:0/3280509347 wait complete. 2024-03-29T11:37:08.127 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.126+0000 7fe364e46640 10 monclient: build_initial_monmap 2024-03-29T11:37:08.127 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.126+0000 7fe364e46640 10 monclient: monmap: 2024-03-29T11:37:08.127 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:37:08.127 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:08.127 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:37:08.127 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:37:08.127 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:08.127 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:08.127 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:37:08.127 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:37:08.128 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:37:08.128 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:08.128 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.127+0000 7fe364e46640 1 librados: starting msgr at 2024-03-29T11:37:08.128 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.127+0000 7fe364e46640 1 librados: starting objecter 2024-03-29T11:37:08.128 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.127+0000 7fe364e46640 1 Processor -- start 2024-03-29T11:37:08.128 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.127+0000 7fe364e46640 1 -- start start 2024-03-29T11:37:08.128 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.127+0000 7fe364e46640 1 librados: setting wanted keys 2024-03-29T11:37:08.128 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.127+0000 7fe364e46640 1 librados: calling monclient init 2024-03-29T11:37:08.128 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.127+0000 7fe364e46640 10 monclient: init 2024-03-29T11:37:08.128 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.128+0000 7fe364e46640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:08.129 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.128+0000 7fe364e46640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:37:08.131 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.129+0000 7fe364e46640 10 monclient(hunting): picked mon.noname-c con 0x7fe3580324b0 addr v1:172.21.15.72:6790/0 2024-03-29T11:37:08.137 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.132+0000 7fe364e46640 10 monclient(hunting): picked mon.noname-a con 0x7fe358038c60 addr v1:172.21.15.72:6789/0 2024-03-29T11:37:08.137 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.135+0000 7fe364e46640 10 monclient(hunting): picked mon.noname-b con 0x7fe3580d11e0 addr v1:172.21.15.89:6789/0 2024-03-29T11:37:08.137 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.135+0000 7fe364e46640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3580be710 con 0x7fe358038c60 2024-03-29T11:37:08.137 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.135+0000 7fe364e46640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3580c5610 con 0x7fe3580d11e0 2024-03-29T11:37:08.137 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.135+0000 7fe364e46640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3580be2f0 con 0x7fe3580324b0 2024-03-29T11:37:08.137 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.135+0000 7fe364e46640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:08.137 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.135+0000 7fe364e46640 10 monclient(hunting): authenticate will time out at 987.423519s 2024-03-29T11:37:08.137 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.135+0000 7fe35f7fe640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7fe3580324b0 0x7fe358169470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:58413/0 (socket says 172.21.15.72:58413) 2024-03-29T11:37:08.137 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.135+0000 7fe35f7fe640 1 -- 172.21.15.72:0/3848493675 learned_addr learned my addr 172.21.15.72:0/3848493675 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:37:08.137 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.135+0000 7fe34bfff640 10 client.?.objecter ms_handle_connect 0x7fe3580324b0 2024-03-29T11:37:08.137 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.135+0000 7fe34bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:08.138 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.136+0000 7fe34bfff640 10 client.?.objecter ms_handle_connect 0x7fe358038c60 2024-03-29T11:37:08.138 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.136+0000 7fe34bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:08.138 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.136+0000 7fe34bfff640 1 -- 172.21.15.72:0/3848493675 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1950810188 0 0) 0x7fe3580be710 con 0x7fe358038c60 2024-03-29T11:37:08.138 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.136+0000 7fe34bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:37:08.138 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.136+0000 7fe34bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:08.138 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.136+0000 7fe34bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:08.138 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.136+0000 7fe34bfff640 10 monclient(hunting): my global_id is 4299 2024-03-29T11:37:08.138 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.136+0000 7fe34bfff640 1 -- 172.21.15.72:0/3848493675 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe344004eb0 con 0x7fe358038c60 2024-03-29T11:37:08.138 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.136+0000 7fe34bfff640 1 -- 172.21.15.72:0/3848493675 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3774405410 0 0) 0x7fe344004eb0 con 0x7fe358038c60 2024-03-29T11:37:08.138 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.136+0000 7fe34bfff640 1 -- 172.21.15.72:0/3848493675 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe344005300 con 0x7fe358038c60 2024-03-29T11:37:08.138 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.136+0000 7fe34bfff640 1 -- 172.21.15.72:0/3848493675 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fe34c0013e0 con 0x7fe358038c60 2024-03-29T11:37:08.138 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.136+0000 7fe34bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:08.138 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.136+0000 7fe34bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:08.138 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.136+0000 7fe34bfff640 10 monclient(hunting): dump: 2024-03-29T11:37:08.138 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:37:08.139 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:08.139 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:08.139 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:08.139 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:08.139 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:08.139 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:37:08.139 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:37:08.139 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:37:08.139 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:08.139 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.136+0000 7fe34bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:08.139 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.137+0000 7fe34bfff640 1 -- 172.21.15.72:0/3848493675 <== mon.0 v1:172.21.15.72:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 46732013 0 0) 0x7fe344005300 con 0x7fe358038c60 2024-03-29T11:37:08.139 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.137+0000 7fe34bfff640 1 -- 172.21.15.72:0/3848493675 >> v1:172.21.15.72:6790/0 conn(0x7fe3580324b0 legacy=0x7fe358169470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:08.139 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.137+0000 7fe34bfff640 1 -- 172.21.15.72:0/3848493675 >> v1:172.21.15.89:6789/0 conn(0x7fe3580d11e0 legacy=0x7fe358174cb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:37:08.141 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.138+0000 7fe34bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:37:08.141 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.138+0000 7fe34bfff640 1 monclient: found mon.a 2024-03-29T11:37:08.141 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.138+0000 7fe34bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:08.141 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.138+0000 7fe34bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:37:08.141 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.138+0000 7fe34bfff640 1 -- 172.21.15.72:0/3848493675 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3580bd040 con 0x7fe358038c60 2024-03-29T11:37:08.141 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.138+0000 7fe34bfff640 10 monclient: _finish_auth 0 2024-03-29T11:37:08.142 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.138+0000 7fe34bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:37:08.142 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.138+0000 7fe34bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:08.142 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.139+0000 7fe364e46640 5 monclient: authenticate success, global_id 4299 2024-03-29T11:37:08.142 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.139+0000 7fe364e46640 10 monclient: _renew_subs 2024-03-29T11:37:08.142 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.139+0000 7fe364e46640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:37:08.142 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.139+0000 7fe364e46640 1 -- 172.21.15.72:0/3848493675 --> v1:172.21.15.72:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe358037290 con 0x7fe358038c60 2024-03-29T11:37:08.142 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.139+0000 7fe364e46640 10 client.4299.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:08.142 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.139+0000 7fe364e46640 10 monclient: _renew_subs 2024-03-29T11:37:08.142 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.139+0000 7fe364e46640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:37:08.142 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.139+0000 7fe364e46640 1 -- 172.21.15.72:0/3848493675 --> v1:172.21.15.72:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe358169b80 con 0x7fe358038c60 2024-03-29T11:37:08.142 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.139+0000 7fe364e46640 1 librados: init done 2024-03-29T11:37:08.143 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.142+0000 7fe34bfff640 1 -- 172.21.15.72:0/3848493675 <== mon.0 v1:172.21.15.72:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe34c000ff0 con 0x7fe358038c60 2024-03-29T11:37:08.143 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.142+0000 7fe34bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:08.143 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.142+0000 7fe364e46640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:08.143 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.142+0000 7fe35dffb640 4 set_mon_vals no callback set 2024-03-29T11:37:08.143 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.143+0000 7fe34bfff640 1 -- 172.21.15.72:0/3848493675 <== mon.0 v1:172.21.15.72:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fe34c0013e0 con 0x7fe358038c60 2024-03-29T11:37:08.143 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.144+0000 7fe364e46640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:08.143 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.144+0000 7fe364e46640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:37:08.143 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.144+0000 7fe364e46640 1 -- 172.21.15.72:0/3848493675 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe358007b90 con 0x7fe358038c60 2024-03-29T11:37:08.143 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.144+0000 7fe34bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:08.143 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.144+0000 7fe34bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:08.144 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.144+0000 7fe34bfff640 10 monclient: dump: 2024-03-29T11:37:08.144 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:37:08.144 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:08.144 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:08.144 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:08.144 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:08.144 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:08.144 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:37:08.144 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:37:08.144 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:37:08.144 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:08.146 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.147+0000 7fe34bfff640 1 -- 172.21.15.72:0/3848493675 <== mon.0 v1:172.21.15.72:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7fe34c019790 con 0x7fe358038c60 2024-03-29T11:37:08.147 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.148+0000 7fe34bfff640 1 -- 172.21.15.72:0/3848493675 <== mon.0 v1:172.21.15.72:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7fe34c083620 con 0x7fe358038c60 2024-03-29T11:37:08.147 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.148+0000 7fe34bfff640 10 client.4299.objecter ms_dispatch 0x7fe360000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:08.147 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.148+0000 7fe34bfff640 3 client.4299.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:08.147 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.148+0000 7fe34bfff640 3 client.4299.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:08.148 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.148+0000 7fe34bfff640 20 client.4299.objecter dump_active .. 0 homeless 2024-03-29T11:37:08.149 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.150+0000 7fe35f7fe640 10 monclient: get_auth_request con 0x7fe34406df80 auth_method 0 2024-03-29T11:37:08.150 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.151+0000 7fe34bfff640 10 client.4299.objecter ms_handle_connect 0x7fe34406df80 2024-03-29T11:37:08.155 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.155+0000 7fe34bfff640 1 -- 172.21.15.72:0/3848493675 <== mon.0 v1:172.21.15.72:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe34c04ed30 con 0x7fe358038c60 2024-03-29T11:37:08.155 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.155+0000 7fe34bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:08.155 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.155+0000 7fe34bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:08.474 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.474+0000 7fe364e46640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:08.474 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.474+0000 7fe364e46640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:08.474 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.474+0000 7fe364e46640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:37:08.474 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.474+0000 7fe364e46640 1 -- 172.21.15.72:0/3848493675 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3580be8a0 con 0x7fe358038c60 2024-03-29T11:37:08.476 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.477+0000 7fe34bfff640 1 -- 172.21.15.72:0/3848493675 <== mon.0 v1:172.21.15.72:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7fe3580be8a0 con 0x7fe358038c60 2024-03-29T11:37:08.476 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.477+0000 7fe34bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:08.476 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.477+0000 7fe34bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:08.477 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:37:08.478 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:08.482 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.482+0000 7fe364e46640 10 librados: watch_flush enter 2024-03-29T11:37:08.482 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.482+0000 7fe364e46640 10 librados: watch_flush exit 2024-03-29T11:37:08.482 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.482+0000 7fe364e46640 20 client.4299.objecter shutdown clearing up homeless session... 2024-03-29T11:37:08.482 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.482+0000 7fe364e46640 10 client.4299.objecter successfully canceled tick 2024-03-29T11:37:08.482 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.482+0000 7fe364e46640 1 -- 172.21.15.72:0/3848493675 >> v1:172.21.15.72:6816/1578866102 conn(0x7fe34406df80 legacy=0x7fe344070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:08.482 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.482+0000 7fe364e46640 10 monclient: shutdown 2024-03-29T11:37:08.482 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.482+0000 7fe364e46640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:08.482 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.482+0000 7fe364e46640 1 -- 172.21.15.72:0/3848493675 >> v1:172.21.15.72:6789/0 conn(0x7fe358038c60 legacy=0x7fe358036640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:08.482 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.482+0000 7fe364e46640 1 -- 172.21.15.72:0/3848493675 shutdown_connections 2024-03-29T11:37:08.482 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.482+0000 7fe364e46640 1 -- 172.21.15.72:0/3848493675 >> 172.21.15.72:0/3848493675 conn(0x7fe358005c20 msgr2=0x7fe3580c1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:08.483 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.483+0000 7fe364e46640 1 -- 172.21.15.72:0/3848493675 shutdown_connections 2024-03-29T11:37:08.483 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.483+0000 7fe364e46640 1 -- 172.21.15.72:0/3848493675 wait complete. 2024-03-29T11:37:08.483 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:08.483+0000 7fe364e46640 1 librados: shutdown 2024-03-29T11:37:08.500 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:15.307 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:15.451 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.450+0000 7f718b214640 10 monclient: get_monmap_and_config 2024-03-29T11:37:15.452 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.450+0000 7f718b214640 10 monclient: build_initial_monmap 2024-03-29T11:37:15.452 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.450+0000 7f718b214640 10 monclient: monmap: 2024-03-29T11:37:15.452 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:37:15.452 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:15.452 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:37:15.452813+0000 2024-03-29T11:37:15.452 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:37:15.452813+0000 2024-03-29T11:37:15.452 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:15.452 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:15.452 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:37:15.452 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:37:15.452 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:37:15.452 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:15.454 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.451+0000 7f718b214640 1 Processor -- start 2024-03-29T11:37:15.454 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.452+0000 7f718b214640 1 -- start start 2024-03-29T11:37:15.455 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.452+0000 7f718b214640 10 monclient: init 2024-03-29T11:37:15.455 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.452+0000 7f718b214640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:15.455 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.452+0000 7f718b214640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:37:15.455 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.452+0000 7f718b214640 10 monclient(hunting): picked mon.noname-c con 0x7f71841457f0 addr v1:172.21.15.89:6789/0 2024-03-29T11:37:15.455 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.452+0000 7f718b214640 10 monclient(hunting): picked mon.noname-a con 0x7f718413f650 addr v1:172.21.15.72:6789/0 2024-03-29T11:37:15.455 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.452+0000 7f718b214640 10 monclient(hunting): picked mon.noname-b con 0x7f71840a9d70 addr v1:172.21.15.72:6790/0 2024-03-29T11:37:15.455 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.452+0000 7f718b214640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71840aaef0 con 0x7f718413f650 2024-03-29T11:37:15.455 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.452+0000 7f718b214640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7184145bd0 con 0x7f71841457f0 2024-03-29T11:37:15.455 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.452+0000 7f718b214640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7184145eb0 con 0x7f71840a9d70 2024-03-29T11:37:15.455 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.452+0000 7f718b214640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:15.455 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.452+0000 7f718b214640 10 monclient(hunting): authenticate will time out at 994.741785s 2024-03-29T11:37:15.455 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.452+0000 7f7183fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f718413f650 0x7f71840b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:40316/0 (socket says 172.21.15.72:40316) 2024-03-29T11:37:15.455 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.452+0000 7f7183fff640 1 -- 172.21.15.72:0/633579500 learned_addr learned my addr 172.21.15.72:0/633579500 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:37:15.455 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.453+0000 7f71837fe640 1 -- 172.21.15.72:0/633579500 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 643988618 0 0) 0x7f7184145eb0 con 0x7f71840a9d70 2024-03-29T11:37:15.456 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.453+0000 7f71837fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:15.456 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.453+0000 7f71837fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:15.456 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.453+0000 7f71837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:15.456 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.453+0000 7f71837fe640 10 monclient(hunting): my global_id is 4301 2024-03-29T11:37:15.456 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.453+0000 7f71837fe640 1 -- 172.21.15.72:0/633579500 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7170003520 con 0x7f71840a9d70 2024-03-29T11:37:15.456 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.453+0000 7f71837fe640 1 -- 172.21.15.72:0/633579500 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 260406982 0 0) 0x7f7170003520 con 0x7f71840a9d70 2024-03-29T11:37:15.456 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.453+0000 7f71837fe640 1 -- 172.21.15.72:0/633579500 >> v1:172.21.15.89:6789/0 conn(0x7f71841457f0 legacy=0x7f718413ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:37:15.457 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.456+0000 7f71837fe640 1 -- 172.21.15.72:0/633579500 >> v1:172.21.15.72:6789/0 conn(0x7f718413f650 legacy=0x7f71840b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:15.457 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.456+0000 7f71837fe640 10 monclient: _finish_hunting 0 2024-03-29T11:37:15.457 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.456+0000 7f71837fe640 1 monclient: found mon.noname-b 2024-03-29T11:37:15.457 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.456+0000 7f71837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:15.457 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.456+0000 7f71837fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.72:6790/0 2024-03-29T11:37:15.457 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.456+0000 7f71837fe640 1 -- 172.21.15.72:0/633579500 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7184146340 con 0x7f71840a9d70 2024-03-29T11:37:15.457 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.456+0000 7f71837fe640 10 monclient: _finish_auth 0 2024-03-29T11:37:15.457 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.456+0000 7f71837fe640 10 monclient: _check_auth_tickets 2024-03-29T11:37:15.457 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.456+0000 7f71837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:15.457 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.456+0000 7f71837fe640 1 -- 172.21.15.72:0/633579500 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f7174001a10 con 0x7f71840a9d70 2024-03-29T11:37:15.457 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.456+0000 7f71837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:15.457 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.456+0000 7f71837fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:15.457 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.456+0000 7f71837fe640 10 monclient: dump: 2024-03-29T11:37:15.458 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:37:15.458 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:15.458 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:15.458 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:15.458 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:15.458 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:15.458 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:37:15.458 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:37:15.458 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:37:15.458 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:15.458 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.456+0000 7f718b214640 5 monclient: authenticate success, global_id 4301 2024-03-29T11:37:15.458 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.456+0000 7f718b214640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:15.458 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.456+0000 7f71837fe640 1 -- 172.21.15.72:0/633579500 <== mon.2 v1:172.21.15.72:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7174001be0 con 0x7f71840a9d70 2024-03-29T11:37:15.458 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.456+0000 7f71837fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:15.458 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.456+0000 7f71837fe640 1 -- 172.21.15.72:0/633579500 <== mon.2 v1:172.21.15.72:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f7174002030 con 0x7f71840a9d70 2024-03-29T11:37:15.459 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.456+0000 7f71837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:15.459 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.456+0000 7f71837fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:15.459 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.456+0000 7f71837fe640 10 monclient: dump: 2024-03-29T11:37:15.459 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:37:15.459 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:15.459 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:15.459 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:15.459 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:15.459 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:15.459 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:37:15.459 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:37:15.459 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:37:15.459 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:15.459 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.456+0000 7f718b214640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:15.459 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.456+0000 7f718b214640 4 set_mon_vals no callback set 2024-03-29T11:37:15.462 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.457+0000 7f718b214640 10 monclient: shutdown 2024-03-29T11:37:15.462 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.457+0000 7f718b214640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:15.463 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.457+0000 7f718b214640 1 -- 172.21.15.72:0/633579500 >> v1:172.21.15.72:6790/0 conn(0x7f71840a9d70 legacy=0x7f718414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:15.463 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.457+0000 7f718b214640 1 -- 172.21.15.72:0/633579500 shutdown_connections 2024-03-29T11:37:15.463 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.457+0000 7f718b214640 1 -- 172.21.15.72:0/633579500 >> 172.21.15.72:0/633579500 conn(0x7f7184066a00 msgr2=0x7f7184066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:15.463 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.458+0000 7f718b214640 1 -- 172.21.15.72:0/633579500 shutdown_connections 2024-03-29T11:37:15.463 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.458+0000 7f718b214640 1 -- 172.21.15.72:0/633579500 wait complete. 2024-03-29T11:37:15.463 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.459+0000 7f718b214640 10 monclient: build_initial_monmap 2024-03-29T11:37:15.463 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.459+0000 7f718b214640 10 monclient: monmap: 2024-03-29T11:37:15.463 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:37:15.463 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:15.463 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:37:15.463 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:37:15.463 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:15.463 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:15.463 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:37:15.463 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:37:15.463 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:37:15.464 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:15.464 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.459+0000 7f718b214640 1 librados: starting msgr at 2024-03-29T11:37:15.464 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.459+0000 7f718b214640 1 librados: starting objecter 2024-03-29T11:37:15.464 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.459+0000 7f718b214640 1 Processor -- start 2024-03-29T11:37:15.464 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.459+0000 7f718b214640 1 -- start start 2024-03-29T11:37:15.464 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.459+0000 7f718b214640 1 librados: setting wanted keys 2024-03-29T11:37:15.464 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.459+0000 7f718b214640 1 librados: calling monclient init 2024-03-29T11:37:15.464 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.459+0000 7f718b214640 10 monclient: init 2024-03-29T11:37:15.464 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.461+0000 7f718b214640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:15.464 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.461+0000 7f718b214640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:37:15.464 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.461+0000 7f718b214640 10 monclient(hunting): picked mon.noname-b con 0x7f71840a9d70 addr v1:172.21.15.89:6789/0 2024-03-29T11:37:15.464 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.461+0000 7f718b214640 10 monclient(hunting): picked mon.noname-a con 0x7f718413f650 addr v1:172.21.15.72:6789/0 2024-03-29T11:37:15.464 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.461+0000 7f718b214640 10 monclient(hunting): picked mon.noname-c con 0x7f71841457f0 addr v1:172.21.15.72:6790/0 2024-03-29T11:37:15.464 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.461+0000 7f718b214640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f718406e890 con 0x7f718413f650 2024-03-29T11:37:15.464 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.461+0000 7f718b214640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7184141fd0 con 0x7f71840a9d70 2024-03-29T11:37:15.465 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.461+0000 7f718b214640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7184142c00 con 0x7f71841457f0 2024-03-29T11:37:15.465 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.461+0000 7f718b214640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:15.465 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.461+0000 7f718b214640 10 monclient(hunting): authenticate will time out at 994.750278s 2024-03-29T11:37:15.465 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.461+0000 7f71897b1640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f71841457f0 0x7f71841db690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:50162/0 (socket says 172.21.15.72:50162) 2024-03-29T11:37:15.465 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.461+0000 7f71897b1640 1 -- 172.21.15.72:0/1503316130 learned_addr learned my addr 172.21.15.72:0/1503316130 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:37:15.465 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.461+0000 7f71817fa640 10 client.?.objecter ms_handle_connect 0x7f71841457f0 2024-03-29T11:37:15.465 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.461+0000 7f71817fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:15.469 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.462+0000 7f71817fa640 10 client.?.objecter ms_handle_connect 0x7f718413f650 2024-03-29T11:37:15.469 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.462+0000 7f71817fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:15.469 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.462+0000 7f71817fa640 1 -- 172.21.15.72:0/1503316130 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1533734524 0 0) 0x7f7184142c00 con 0x7f71841457f0 2024-03-29T11:37:15.469 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.462+0000 7f71817fa640 10 monclient(hunting): _negotiate 2024-03-29T11:37:15.470 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.462+0000 7f71817fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:15.470 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.462+0000 7f71817fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:15.470 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.462+0000 7f71817fa640 10 monclient(hunting): my global_id is 4304 2024-03-29T11:37:15.470 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.462+0000 7f71817fa640 1 -- 172.21.15.72:0/1503316130 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7164004eb0 con 0x7f71841457f0 2024-03-29T11:37:15.470 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.463+0000 7f71817fa640 1 -- 172.21.15.72:0/1503316130 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3813378669 0 0) 0x7f7164004eb0 con 0x7f71841457f0 2024-03-29T11:37:15.470 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.463+0000 7f71817fa640 1 -- 172.21.15.72:0/1503316130 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7184142c00 con 0x7f71841457f0 2024-03-29T11:37:15.470 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.463+0000 7f71817fa640 1 -- 172.21.15.72:0/1503316130 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f7174002030 con 0x7f71841457f0 2024-03-29T11:37:15.470 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.463+0000 7f71817fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:15.470 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.463+0000 7f71817fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:15.470 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.463+0000 7f71817fa640 10 monclient(hunting): dump: 2024-03-29T11:37:15.470 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:37:15.470 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:15.470 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:15.470 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:15.471 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:15.471 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:15.471 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:37:15.471 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:37:15.471 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:37:15.471 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:15.471 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.463+0000 7f71817fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:15.471 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.464+0000 7f71817fa640 1 -- 172.21.15.72:0/1503316130 <== mon.2 v1:172.21.15.72:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3777763533 0 0) 0x7f7184142c00 con 0x7f71841457f0 2024-03-29T11:37:15.471 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.464+0000 7f71817fa640 1 -- 172.21.15.72:0/1503316130 >> v1:172.21.15.89:6789/0 conn(0x7f71840a9d70 legacy=0x7f7184141690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:37:15.471 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.464+0000 7f71817fa640 1 -- 172.21.15.72:0/1503316130 >> v1:172.21.15.72:6789/0 conn(0x7f718413f650 legacy=0x7f71841424f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:15.471 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.464+0000 7f71817fa640 10 monclient: _finish_hunting 0 2024-03-29T11:37:15.471 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.464+0000 7f71817fa640 1 monclient: found mon.c 2024-03-29T11:37:15.471 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.464+0000 7f71817fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:15.471 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.464+0000 7f71817fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:37:15.471 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.464+0000 7f71817fa640 1 -- 172.21.15.72:0/1503316130 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7184144710 con 0x7f71841457f0 2024-03-29T11:37:15.472 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.464+0000 7f71817fa640 10 monclient: _finish_auth 0 2024-03-29T11:37:15.472 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.464+0000 7f71817fa640 10 monclient: _check_auth_tickets 2024-03-29T11:37:15.472 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.464+0000 7f71817fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:15.472 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.464+0000 7f718b214640 5 monclient: authenticate success, global_id 4304 2024-03-29T11:37:15.472 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.464+0000 7f718b214640 10 monclient: _renew_subs 2024-03-29T11:37:15.472 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.464+0000 7f718b214640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:37:15.472 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.464+0000 7f718b214640 1 -- 172.21.15.72:0/1503316130 --> v1:172.21.15.72:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71841449e0 con 0x7f71841457f0 2024-03-29T11:37:15.472 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.465+0000 7f718b214640 10 client.4304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:15.472 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.465+0000 7f71817fa640 1 -- 172.21.15.72:0/1503316130 <== mon.2 v1:172.21.15.72:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7174003ce0 con 0x7f71841457f0 2024-03-29T11:37:15.472 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.465+0000 7f71817fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:15.472 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.465+0000 7f71817fa640 1 -- 172.21.15.72:0/1503316130 <== mon.2 v1:172.21.15.72:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f7174004130 con 0x7f71841457f0 2024-03-29T11:37:15.472 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.465+0000 7f71817fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:15.472 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.465+0000 7f71817fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:15.472 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.465+0000 7f71817fa640 10 monclient: dump: 2024-03-29T11:37:15.472 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:37:15.472 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:15.473 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:15.473 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:15.473 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:15.473 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:15.473 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:37:15.473 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:37:15.473 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:37:15.473 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:15.473 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.465+0000 7f7182ffd640 4 set_mon_vals no callback set 2024-03-29T11:37:15.473 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.465+0000 7f718b214640 10 monclient: _renew_subs 2024-03-29T11:37:15.473 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.465+0000 7f718b214640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:37:15.473 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.465+0000 7f718b214640 1 -- 172.21.15.72:0/1503316130 --> v1:172.21.15.72:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f718413ff80 con 0x7f71841457f0 2024-03-29T11:37:15.473 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.465+0000 7f718b214640 1 librados: init done 2024-03-29T11:37:15.473 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.467+0000 7f71817fa640 1 -- 172.21.15.72:0/1503316130 <== mon.2 v1:172.21.15.72:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f7174004290 con 0x7f71841457f0 2024-03-29T11:37:15.473 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.468+0000 7f7188fb0640 10 monclient: get_auth_request con 0x7f716406dcc0 auth_method 0 2024-03-29T11:37:15.473 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.468+0000 7f718b214640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:15.474 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.468+0000 7f718b214640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:15.474 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.468+0000 7f718b214640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:37:15.474 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.468+0000 7f718b214640 1 -- 172.21.15.72:0/1503316130 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7184142160 con 0x7f71841457f0 2024-03-29T11:37:15.474 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.469+0000 7f71817fa640 1 -- 172.21.15.72:0/1503316130 <== mon.2 v1:172.21.15.72:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f7174084fb0 con 0x7f71841457f0 2024-03-29T11:37:15.474 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.469+0000 7f71817fa640 10 client.4304.objecter ms_dispatch 0x7f7184000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:15.474 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.469+0000 7f71817fa640 3 client.4304.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:15.474 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.469+0000 7f71817fa640 3 client.4304.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:15.474 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.469+0000 7f71817fa640 20 client.4304.objecter dump_active .. 0 homeless 2024-03-29T11:37:15.474 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.469+0000 7f71817fa640 10 client.4304.objecter ms_handle_connect 0x7f716406dcc0 2024-03-29T11:37:15.477 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.476+0000 7f71817fa640 1 -- 172.21.15.72:0/1503316130 <== mon.2 v1:172.21.15.72:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7174050890 con 0x7f71841457f0 2024-03-29T11:37:15.477 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.476+0000 7f71817fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:15.477 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.476+0000 7f71817fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:15.795 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.795+0000 7f718b214640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:15.795 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.795+0000 7f718b214640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:15.796 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.795+0000 7f718b214640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:37:15.796 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.795+0000 7f718b214640 1 -- 172.21.15.72:0/1503316130 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7184075f10 con 0x7f71841457f0 2024-03-29T11:37:15.797 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.796+0000 7f71817fa640 1 -- 172.21.15.72:0/1503316130 <== mon.2 v1:172.21.15.72:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f71740527c0 con 0x7f71841457f0 2024-03-29T11:37:15.797 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.796+0000 7f71817fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:15.797 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.796+0000 7f71817fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:15.797 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:37:15.797 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:15.801 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.800+0000 7f718b214640 10 librados: watch_flush enter 2024-03-29T11:37:15.801 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.800+0000 7f718b214640 10 librados: watch_flush exit 2024-03-29T11:37:15.801 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.800+0000 7f718b214640 20 client.4304.objecter shutdown clearing up homeless session... 2024-03-29T11:37:15.801 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.800+0000 7f718b214640 10 client.4304.objecter successfully canceled tick 2024-03-29T11:37:15.801 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.800+0000 7f718b214640 1 -- 172.21.15.72:0/1503316130 >> v1:172.21.15.72:6816/1578866102 conn(0x7f716406dcc0 legacy=0x7f7164070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:15.801 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.800+0000 7f718b214640 10 monclient: shutdown 2024-03-29T11:37:15.801 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.800+0000 7f718b214640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:15.801 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.800+0000 7f718b214640 1 -- 172.21.15.72:0/1503316130 >> v1:172.21.15.72:6790/0 conn(0x7f71841457f0 legacy=0x7f71841db690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:15.801 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.800+0000 7f718b214640 1 -- 172.21.15.72:0/1503316130 shutdown_connections 2024-03-29T11:37:15.801 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.800+0000 7f718b214640 1 -- 172.21.15.72:0/1503316130 >> 172.21.15.72:0/1503316130 conn(0x7f7184066a00 msgr2=0x7f7184066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:15.801 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.800+0000 7f718b214640 1 -- 172.21.15.72:0/1503316130 shutdown_connections 2024-03-29T11:37:15.802 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.800+0000 7f718b214640 1 -- 172.21.15.72:0/1503316130 wait complete. 2024-03-29T11:37:15.802 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:15.800+0000 7f718b214640 1 librados: shutdown 2024-03-29T11:37:15.815 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:22.621 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:22.778 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.778+0000 7f7bbaced640 10 monclient: get_monmap_and_config 2024-03-29T11:37:22.778 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.778+0000 7f7bbaced640 10 monclient: build_initial_monmap 2024-03-29T11:37:22.779 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.778+0000 7f7bbaced640 10 monclient: monmap: 2024-03-29T11:37:22.779 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:37:22.779 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:22.779 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:37:22.779439+0000 2024-03-29T11:37:22.779 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:37:22.779439+0000 2024-03-29T11:37:22.779 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:22.779 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:22.779 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:37:22.779 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:37:22.779 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:37:22.779 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:22.780 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.779+0000 7f7bbaced640 1 Processor -- start 2024-03-29T11:37:22.780 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.779+0000 7f7bbaced640 1 -- start start 2024-03-29T11:37:22.780 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.779+0000 7f7bbaced640 10 monclient: init 2024-03-29T11:37:22.780 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.780+0000 7f7bbaced640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:22.780 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.780+0000 7f7bbaced640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:37:22.780 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.780+0000 7f7bbaced640 10 monclient(hunting): picked mon.noname-a con 0x7f7bb41457f0 addr v1:172.21.15.72:6789/0 2024-03-29T11:37:22.780 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.780+0000 7f7bbaced640 10 monclient(hunting): picked mon.noname-b con 0x7f7bb413f650 addr v1:172.21.15.72:6790/0 2024-03-29T11:37:22.781 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.780+0000 7f7bbaced640 10 monclient(hunting): picked mon.noname-c con 0x7f7bb40a9d70 addr v1:172.21.15.89:6789/0 2024-03-29T11:37:22.781 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.780+0000 7f7bbaced640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb40aaef0 con 0x7f7bb41457f0 2024-03-29T11:37:22.781 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.780+0000 7f7bbaced640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb4145bd0 con 0x7f7bb40a9d70 2024-03-29T11:37:22.781 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.780+0000 7f7bbaced640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb4145eb0 con 0x7f7bb413f650 2024-03-29T11:37:22.781 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.780+0000 7f7bbaced640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:22.781 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.780+0000 7f7bbaced640 10 monclient(hunting): authenticate will time out at 1002.068605s 2024-03-29T11:37:22.781 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.780+0000 7f7bb928a640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f7bb40a9d70 0x7f7bb414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:47350/0 (socket says 172.21.15.72:47350) 2024-03-29T11:37:22.781 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.780+0000 7f7bb928a640 1 -- 172.21.15.72:0/2005201613 learned_addr learned my addr 172.21.15.72:0/2005201613 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:37:22.784 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.781+0000 7f7bb37fe640 1 -- 172.21.15.72:0/2005201613 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2921509653 0 0) 0x7f7bb4145bd0 con 0x7f7bb40a9d70 2024-03-29T11:37:22.784 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.781+0000 7f7bb37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:22.784 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.781+0000 7f7bb37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:22.784 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.781+0000 7f7bb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:22.784 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.781+0000 7f7bb37fe640 10 monclient(hunting): my global_id is 4303 2024-03-29T11:37:22.784 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.781+0000 7f7bb37fe640 1 -- 172.21.15.72:0/2005201613 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ba0003520 con 0x7f7bb40a9d70 2024-03-29T11:37:22.784 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.781+0000 7f7bb37fe640 1 -- 172.21.15.72:0/2005201613 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1959985571 0 0) 0x7f7bb4145eb0 con 0x7f7bb413f650 2024-03-29T11:37:22.784 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.781+0000 7f7bb37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:22.784 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.781+0000 7f7bb37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:22.785 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.781+0000 7f7bb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:22.785 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.781+0000 7f7bb37fe640 10 monclient(hunting): my global_id is 4307 2024-03-29T11:37:22.785 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.781+0000 7f7bb37fe640 1 -- 172.21.15.72:0/2005201613 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bb4145bd0 con 0x7f7bb413f650 2024-03-29T11:37:22.785 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.782+0000 7f7bb37fe640 1 -- 172.21.15.72:0/2005201613 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2875792680 0 0) 0x7f7ba0003520 con 0x7f7bb40a9d70 2024-03-29T11:37:22.785 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.782+0000 7f7bb37fe640 1 -- 172.21.15.72:0/2005201613 >> v1:172.21.15.72:6790/0 conn(0x7f7bb413f650 legacy=0x7f7bb40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:22.785 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.782+0000 7f7bb37fe640 1 -- 172.21.15.72:0/2005201613 >> v1:172.21.15.72:6789/0 conn(0x7f7bb41457f0 legacy=0x7f7bb413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:22.785 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.782+0000 7f7bb37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:37:22.785 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.782+0000 7f7bb37fe640 1 monclient: found mon.noname-c 2024-03-29T11:37:22.785 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.782+0000 7f7bb37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:22.785 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.782+0000 7f7bb37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.89:6789/0 2024-03-29T11:37:22.786 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.782+0000 7f7bb37fe640 1 -- 172.21.15.72:0/2005201613 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb4146340 con 0x7f7bb40a9d70 2024-03-29T11:37:22.786 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.782+0000 7f7bb37fe640 10 monclient: _finish_auth 0 2024-03-29T11:37:22.786 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.782+0000 7f7bb37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:37:22.786 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.782+0000 7f7bb37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:22.786 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.782+0000 7f7bb37fe640 1 -- 172.21.15.72:0/2005201613 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f7bac001690 con 0x7f7bb40a9d70 2024-03-29T11:37:22.786 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.782+0000 7f7bb37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:22.786 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.782+0000 7f7bb37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:22.786 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.782+0000 7f7bb37fe640 10 monclient: dump: 2024-03-29T11:37:22.786 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:37:22.787 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:22.787 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:22.787 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:22.787 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:22.787 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:22.787 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:37:22.787 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:37:22.787 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:37:22.787 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:22.787 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.782+0000 7f7bbaced640 5 monclient: authenticate success, global_id 4303 2024-03-29T11:37:22.787 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.782+0000 7f7bbaced640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:22.787 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.783+0000 7f7bb37fe640 1 -- 172.21.15.72:0/2005201613 <== mon.1 v1:172.21.15.89:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7bac0018b0 con 0x7f7bb40a9d70 2024-03-29T11:37:22.788 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.783+0000 7f7bb37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:22.788 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.783+0000 7f7bbaced640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:22.788 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.783+0000 7f7bbaced640 4 set_mon_vals no callback set 2024-03-29T11:37:22.788 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.783+0000 7f7bb37fe640 1 -- 172.21.15.72:0/2005201613 <== mon.1 v1:172.21.15.89:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f7bac001c40 con 0x7f7bb40a9d70 2024-03-29T11:37:22.788 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.783+0000 7f7bb37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:22.788 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.783+0000 7f7bb37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:22.788 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.783+0000 7f7bb37fe640 10 monclient: dump: 2024-03-29T11:37:22.788 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:37:22.788 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:22.788 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:22.788 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:22.788 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:22.788 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:22.788 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:37:22.789 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:37:22.789 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:37:22.789 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:22.789 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.784+0000 7f7bbaced640 10 monclient: shutdown 2024-03-29T11:37:22.789 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.784+0000 7f7bbaced640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:22.789 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.784+0000 7f7bbaced640 1 -- 172.21.15.72:0/2005201613 >> v1:172.21.15.89:6789/0 conn(0x7f7bb40a9d70 legacy=0x7f7bb414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:22.789 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.784+0000 7f7bbaced640 1 -- 172.21.15.72:0/2005201613 shutdown_connections 2024-03-29T11:37:22.789 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.784+0000 7f7bbaced640 1 -- 172.21.15.72:0/2005201613 >> 172.21.15.72:0/2005201613 conn(0x7f7bb4066a00 msgr2=0x7f7bb4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:22.789 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.784+0000 7f7bbaced640 1 -- 172.21.15.72:0/2005201613 shutdown_connections 2024-03-29T11:37:22.789 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.784+0000 7f7bbaced640 1 -- 172.21.15.72:0/2005201613 wait complete. 2024-03-29T11:37:22.793 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.785+0000 7f7bbaced640 10 monclient: build_initial_monmap 2024-03-29T11:37:22.793 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.785+0000 7f7bbaced640 10 monclient: monmap: 2024-03-29T11:37:22.793 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:37:22.793 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:22.793 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:37:22.793 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:37:22.793 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:22.793 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:22.794 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:37:22.794 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:37:22.794 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:37:22.794 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:22.794 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.785+0000 7f7bbaced640 1 librados: starting msgr at 2024-03-29T11:37:22.794 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.785+0000 7f7bbaced640 1 librados: starting objecter 2024-03-29T11:37:22.794 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.785+0000 7f7bbaced640 1 Processor -- start 2024-03-29T11:37:22.794 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.786+0000 7f7bbaced640 1 -- start start 2024-03-29T11:37:22.794 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.786+0000 7f7bbaced640 1 librados: setting wanted keys 2024-03-29T11:37:22.794 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.786+0000 7f7bbaced640 1 librados: calling monclient init 2024-03-29T11:37:22.794 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.786+0000 7f7bbaced640 10 monclient: init 2024-03-29T11:37:22.794 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.787+0000 7f7bbaced640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:22.794 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.787+0000 7f7bbaced640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:37:22.794 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.787+0000 7f7bbaced640 10 monclient(hunting): picked mon.noname-b con 0x7f7bb40a9d70 addr v1:172.21.15.89:6789/0 2024-03-29T11:37:22.794 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.787+0000 7f7bb8a89640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f7bb40a9d70 0x7f7bb40adb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:47542/0 (socket says 172.21.15.72:47542) 2024-03-29T11:37:22.795 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.787+0000 7f7bb8a89640 1 -- 172.21.15.72:0/1267121365 learned_addr learned my addr 172.21.15.72:0/1267121365 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:37:22.795 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.787+0000 7f7bbaced640 10 monclient(hunting): picked mon.noname-a con 0x7f7bb413f650 addr v1:172.21.15.72:6789/0 2024-03-29T11:37:22.795 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.787+0000 7f7bbaced640 10 monclient(hunting): picked mon.noname-c con 0x7f7bb41457f0 addr v1:172.21.15.72:6790/0 2024-03-29T11:37:22.795 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.787+0000 7f7bbaced640 1 -- 172.21.15.72:0/1267121365 --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb40aff70 con 0x7f7bb413f650 2024-03-29T11:37:22.795 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.787+0000 7f7bbaced640 1 -- 172.21.15.72:0/1267121365 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb40b01e0 con 0x7f7bb40a9d70 2024-03-29T11:37:22.795 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.787+0000 7f7bb17fa640 10 client.?.objecter ms_handle_connect 0x7f7bb40a9d70 2024-03-29T11:37:22.795 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.787+0000 7f7bb17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:22.795 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.787+0000 7f7bbaced640 1 -- 172.21.15.72:0/1267121365 --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bb41d7880 con 0x7f7bb41457f0 2024-03-29T11:37:22.795 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.787+0000 7f7bbaced640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:22.795 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.787+0000 7f7bbaced640 10 monclient(hunting): authenticate will time out at 1002.075754s 2024-03-29T11:37:22.795 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.787+0000 7f7bb17fa640 10 client.?.objecter ms_handle_connect 0x7f7bb413f650 2024-03-29T11:37:22.795 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.787+0000 7f7bb17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:22.795 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.788+0000 7f7bb17fa640 10 client.?.objecter ms_handle_connect 0x7f7bb41457f0 2024-03-29T11:37:22.795 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.788+0000 7f7bb17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:22.795 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.788+0000 7f7bb17fa640 1 -- 172.21.15.72:0/1267121365 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1746922543 0 0) 0x7f7bb40b01e0 con 0x7f7bb40a9d70 2024-03-29T11:37:22.796 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.788+0000 7f7bb17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:37:22.796 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.788+0000 7f7bb17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:22.796 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.788+0000 7f7bb17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:22.796 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.788+0000 7f7bb17fa640 10 monclient(hunting): my global_id is 4306 2024-03-29T11:37:22.796 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.788+0000 7f7bb17fa640 1 -- 172.21.15.72:0/1267121365 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b94004eb0 con 0x7f7bb40a9d70 2024-03-29T11:37:22.796 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.788+0000 7f7bb17fa640 1 -- 172.21.15.72:0/1267121365 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1269843879 0 0) 0x7f7bb40aff70 con 0x7f7bb413f650 2024-03-29T11:37:22.796 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.788+0000 7f7bb17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:37:22.796 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.788+0000 7f7bb17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:22.796 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.788+0000 7f7bb17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:22.796 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.788+0000 7f7bb17fa640 10 monclient(hunting): my global_id is 4311 2024-03-29T11:37:22.796 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.788+0000 7f7bb17fa640 1 -- 172.21.15.72:0/1267121365 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bb40b01e0 con 0x7f7bb413f650 2024-03-29T11:37:22.796 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.789+0000 7f7bb17fa640 1 -- 172.21.15.72:0/1267121365 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 457134940 0 0) 0x7f7b94004eb0 con 0x7f7bb40a9d70 2024-03-29T11:37:22.796 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.789+0000 7f7bb17fa640 1 -- 172.21.15.72:0/1267121365 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7bb40aff70 con 0x7f7bb40a9d70 2024-03-29T11:37:22.796 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.789+0000 7f7bb17fa640 1 -- 172.21.15.72:0/1267121365 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f7ba4001cf0 con 0x7f7bb40a9d70 2024-03-29T11:37:22.796 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.789+0000 7f7bb17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:22.796 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.789+0000 7f7bb17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:22.797 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.789+0000 7f7bb17fa640 10 monclient(hunting): dump: 2024-03-29T11:37:22.797 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:37:22.797 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:22.797 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:22.797 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:22.797 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:22.797 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:22.797 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:37:22.797 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:37:22.797 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:37:22.797 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:22.797 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.789+0000 7f7bb17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:22.797 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.789+0000 7f7bb17fa640 1 -- 172.21.15.72:0/1267121365 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1757483540 0 0) 0x7f7bb41d7880 con 0x7f7bb41457f0 2024-03-29T11:37:22.797 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.789+0000 7f7bb17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:37:22.797 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.789+0000 7f7bb17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:22.797 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.789+0000 7f7bb17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:22.798 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.789+0000 7f7bb17fa640 10 monclient(hunting): my global_id is 4310 2024-03-29T11:37:22.798 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.789+0000 7f7bb17fa640 1 -- 172.21.15.72:0/1267121365 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b94004eb0 con 0x7f7bb41457f0 2024-03-29T11:37:22.798 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.789+0000 7f7bb17fa640 1 -- 172.21.15.72:0/1267121365 <== mon.1 v1:172.21.15.89:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3197763625 0 0) 0x7f7bb40aff70 con 0x7f7bb40a9d70 2024-03-29T11:37:22.798 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.789+0000 7f7bb17fa640 1 -- 172.21.15.72:0/1267121365 >> v1:172.21.15.72:6790/0 conn(0x7f7bb41457f0 legacy=0x7f7bb41e2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:22.798 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.789+0000 7f7bb17fa640 1 -- 172.21.15.72:0/1267121365 >> v1:172.21.15.72:6789/0 conn(0x7f7bb413f650 legacy=0x7f7bb40ae5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:22.798 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.789+0000 7f7bb17fa640 10 monclient: _finish_hunting 0 2024-03-29T11:37:22.798 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.789+0000 7f7bb17fa640 1 monclient: found mon.b 2024-03-29T11:37:22.798 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.789+0000 7f7bb17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:22.798 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.789+0000 7f7bb17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:37:22.798 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.789+0000 7f7bb17fa640 1 -- 172.21.15.72:0/1267121365 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bb40b1390 con 0x7f7bb40a9d70 2024-03-29T11:37:22.798 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.789+0000 7f7bb17fa640 10 monclient: _finish_auth 0 2024-03-29T11:37:22.798 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.789+0000 7f7bb17fa640 10 monclient: _check_auth_tickets 2024-03-29T11:37:22.798 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.789+0000 7f7bb17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:22.798 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.789+0000 7f7bbaced640 5 monclient: authenticate success, global_id 4306 2024-03-29T11:37:22.798 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.789+0000 7f7bbaced640 10 monclient: _renew_subs 2024-03-29T11:37:22.798 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.789+0000 7f7bbaced640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:37:22.799 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.789+0000 7f7bbaced640 1 -- 172.21.15.72:0/1267121365 --> v1:172.21.15.89:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bb41d72a0 con 0x7f7bb40a9d70 2024-03-29T11:37:22.799 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.789+0000 7f7bbaced640 10 client.4306.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:22.799 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.789+0000 7f7bbaced640 10 monclient: _renew_subs 2024-03-29T11:37:22.799 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.789+0000 7f7bbaced640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:37:22.799 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.789+0000 7f7bbaced640 1 -- 172.21.15.72:0/1267121365 --> v1:172.21.15.89:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bb40b0b50 con 0x7f7bb40a9d70 2024-03-29T11:37:22.799 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.789+0000 7f7bbaced640 1 librados: init done 2024-03-29T11:37:22.799 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.789+0000 7f7bb17fa640 1 -- 172.21.15.72:0/1267121365 <== mon.1 v1:172.21.15.89:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ba4001e50 con 0x7f7bb40a9d70 2024-03-29T11:37:22.799 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.789+0000 7f7bb17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:22.799 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.789+0000 7f7bb17fa640 1 -- 172.21.15.72:0/1267121365 <== mon.1 v1:172.21.15.89:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f7ba4003c40 con 0x7f7bb40a9d70 2024-03-29T11:37:22.799 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.789+0000 7f7bb17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:22.799 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.789+0000 7f7bb17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:22.799 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.789+0000 7f7bb17fa640 10 monclient: dump: 2024-03-29T11:37:22.799 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:37:22.799 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:22.799 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:22.800 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:22.800 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:22.800 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:22.800 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:37:22.800 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:37:22.800 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:37:22.800 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:22.800 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.789+0000 7f7bb2ffd640 4 set_mon_vals no callback set 2024-03-29T11:37:22.800 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.791+0000 7f7bbaced640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:22.800 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.791+0000 7f7bbaced640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:22.800 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.791+0000 7f7bbaced640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:37:22.800 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.791+0000 7f7bbaced640 1 -- 172.21.15.72:0/1267121365 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b98005080 con 0x7f7bb40a9d70 2024-03-29T11:37:22.800 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.792+0000 7f7bb17fa640 1 -- 172.21.15.72:0/1267121365 <== mon.1 v1:172.21.15.89:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f7ba4003da0 con 0x7f7bb40a9d70 2024-03-29T11:37:22.800 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.793+0000 7f7bb3fff640 10 monclient: get_auth_request con 0x7f7b9406de40 auth_method 0 2024-03-29T11:37:22.800 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.793+0000 7f7bb17fa640 1 -- 172.21.15.72:0/1267121365 <== mon.1 v1:172.21.15.89:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f7ba4084d90 con 0x7f7bb40a9d70 2024-03-29T11:37:22.800 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.793+0000 7f7bb17fa640 10 client.4306.objecter ms_dispatch 0x7f7bb4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:22.801 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.793+0000 7f7bb17fa640 3 client.4306.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:22.801 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.793+0000 7f7bb17fa640 3 client.4306.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:22.801 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.793+0000 7f7bb17fa640 20 client.4306.objecter dump_active .. 0 homeless 2024-03-29T11:37:22.801 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.793+0000 7f7bb17fa640 10 client.4306.objecter ms_handle_connect 0x7f7b9406de40 2024-03-29T11:37:22.801 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.800+0000 7f7bb17fa640 1 -- 172.21.15.72:0/1267121365 <== mon.1 v1:172.21.15.89:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7ba4050670 con 0x7f7bb40a9d70 2024-03-29T11:37:22.801 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.800+0000 7f7bb17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:22.801 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:22.800+0000 7f7bb17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:23.170 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:23.169+0000 7f7bbaced640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:23.170 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:23.169+0000 7f7bbaced640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:23.170 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:23.170+0000 7f7bbaced640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:37:23.170 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:23.170+0000 7f7bbaced640 1 -- 172.21.15.72:0/1267121365 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b98005330 con 0x7f7bb40a9d70 2024-03-29T11:37:23.171 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:23.171+0000 7f7bb17fa640 1 -- 172.21.15.72:0/1267121365 <== mon.1 v1:172.21.15.89:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f7ba40525a0 con 0x7f7bb40a9d70 2024-03-29T11:37:23.171 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:23.171+0000 7f7bb17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:23.171 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:23.171+0000 7f7bb17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:23.171 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:37:23.172 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:23.173 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:23.173+0000 7f7bbaced640 10 librados: watch_flush enter 2024-03-29T11:37:23.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:23.173+0000 7f7bbaced640 10 librados: watch_flush exit 2024-03-29T11:37:23.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:23.173+0000 7f7bbaced640 20 client.4306.objecter shutdown clearing up homeless session... 2024-03-29T11:37:23.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:23.173+0000 7f7bbaced640 10 client.4306.objecter successfully canceled tick 2024-03-29T11:37:23.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:23.173+0000 7f7bbaced640 1 -- 172.21.15.72:0/1267121365 >> v1:172.21.15.72:6816/1578866102 conn(0x7f7b9406de40 legacy=0x7f7b94070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:23.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:23.173+0000 7f7bbaced640 10 monclient: shutdown 2024-03-29T11:37:23.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:23.173+0000 7f7bbaced640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:23.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:23.173+0000 7f7bbaced640 1 -- 172.21.15.72:0/1267121365 >> v1:172.21.15.89:6789/0 conn(0x7f7bb40a9d70 legacy=0x7f7bb40adb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:23.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:23.173+0000 7f7bbaced640 1 -- 172.21.15.72:0/1267121365 shutdown_connections 2024-03-29T11:37:23.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:23.173+0000 7f7bbaced640 1 -- 172.21.15.72:0/1267121365 >> 172.21.15.72:0/1267121365 conn(0x7f7bb4066a00 msgr2=0x7f7bb40b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:23.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:23.174+0000 7f7bbaced640 1 -- 172.21.15.72:0/1267121365 shutdown_connections 2024-03-29T11:37:23.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:23.174+0000 7f7bbaced640 1 -- 172.21.15.72:0/1267121365 wait complete. 2024-03-29T11:37:23.175 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:23.174+0000 7f7bbaced640 1 librados: shutdown 2024-03-29T11:37:23.187 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:29.994 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:30.139 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.139+0000 7f5f38afe640 10 monclient: get_monmap_and_config 2024-03-29T11:37:30.139 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.139+0000 7f5f38afe640 10 monclient: build_initial_monmap 2024-03-29T11:37:30.140 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.139+0000 7f5f38afe640 10 monclient: monmap: 2024-03-29T11:37:30.140 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:37:30.140 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:30.140 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:37:30.140875+0000 2024-03-29T11:37:30.140 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:37:30.140875+0000 2024-03-29T11:37:30.140 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:30.140 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:30.140 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:37:30.140 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:37:30.140 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:37:30.140 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:30.141 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.141+0000 7f5f38afe640 1 Processor -- start 2024-03-29T11:37:30.141 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.141+0000 7f5f38afe640 1 -- start start 2024-03-29T11:37:30.142 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.141+0000 7f5f38afe640 10 monclient: init 2024-03-29T11:37:30.142 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.141+0000 7f5f38afe640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:30.142 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.141+0000 7f5f38afe640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:37:30.142 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.141+0000 7f5f38afe640 10 monclient(hunting): picked mon.noname-b con 0x7f5f340b4820 addr v1:172.21.15.72:6790/0 2024-03-29T11:37:30.142 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.141+0000 7f5f38afe640 10 monclient(hunting): picked mon.noname-c con 0x7f5f340a9ce0 addr v1:172.21.15.89:6789/0 2024-03-29T11:37:30.142 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.141+0000 7f5f38afe640 10 monclient(hunting): picked mon.noname-a con 0x7f5f340b3cd0 addr v1:172.21.15.72:6789/0 2024-03-29T11:37:30.142 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.141+0000 7f5f38afe640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f340b4180 con 0x7f5f340b3cd0 2024-03-29T11:37:30.142 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.141+0000 7f5f38afe640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f340aaf60 con 0x7f5f340a9ce0 2024-03-29T11:37:30.142 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.141+0000 7f5f38afe640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f3413c200 con 0x7f5f340b4820 2024-03-29T11:37:30.142 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.141+0000 7f5f38afe640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:30.142 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.141+0000 7f5f38afe640 10 monclient(hunting): authenticate will time out at 1009.429987s 2024-03-29T11:37:30.143 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.143+0000 7f5f32ffd640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f5f340a9ce0 0x7f5f34145770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:36682/0 (socket says 172.21.15.72:36682) 2024-03-29T11:37:30.143 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.143+0000 7f5f32ffd640 1 -- 172.21.15.72:0/3749287534 learned_addr learned my addr 172.21.15.72:0/3749287534 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:37:30.145 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.144+0000 7f5f327fc640 1 -- 172.21.15.72:0/3749287534 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4011312109 0 0) 0x7f5f340aaf60 con 0x7f5f340a9ce0 2024-03-29T11:37:30.145 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.144+0000 7f5f327fc640 10 monclient(hunting): _negotiate 2024-03-29T11:37:30.145 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.144+0000 7f5f327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:30.145 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.144+0000 7f5f327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:30.145 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.144+0000 7f5f327fc640 10 monclient(hunting): my global_id is 4309 2024-03-29T11:37:30.145 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.144+0000 7f5f327fc640 1 -- 172.21.15.72:0/3749287534 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f20003520 con 0x7f5f340a9ce0 2024-03-29T11:37:30.145 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.144+0000 7f5f327fc640 1 -- 172.21.15.72:0/3749287534 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4167663673 0 0) 0x7f5f2c001490 con 0x7f5f340b3cd0 2024-03-29T11:37:30.145 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.144+0000 7f5f327fc640 10 monclient(hunting): _negotiate 2024-03-29T11:37:30.145 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.144+0000 7f5f327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:30.145 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.144+0000 7f5f327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:30.146 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.144+0000 7f5f327fc640 10 monclient(hunting): my global_id is 4314 2024-03-29T11:37:30.146 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.144+0000 7f5f327fc640 1 -- 172.21.15.72:0/3749287534 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f340aaf60 con 0x7f5f340b3cd0 2024-03-29T11:37:30.146 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.145+0000 7f5f327fc640 1 -- 172.21.15.72:0/3749287534 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3128997559 0 0) 0x7f5f340aaf60 con 0x7f5f340b3cd0 2024-03-29T11:37:30.146 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.145+0000 7f5f327fc640 1 -- 172.21.15.72:0/3749287534 >> v1:172.21.15.72:6790/0 conn(0x7f5f340b4820 legacy=0x7f5f3413baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:37:30.146 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.145+0000 7f5f327fc640 1 -- 172.21.15.72:0/3749287534 >> v1:172.21.15.89:6789/0 conn(0x7f5f340a9ce0 legacy=0x7f5f34145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:30.146 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.145+0000 7f5f327fc640 10 monclient: _finish_hunting 0 2024-03-29T11:37:30.146 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.145+0000 7f5f327fc640 1 monclient: found mon.noname-a 2024-03-29T11:37:30.146 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.145+0000 7f5f327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:30.146 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.145+0000 7f5f327fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.72:6789/0 2024-03-29T11:37:30.146 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.145+0000 7f5f327fc640 1 -- 172.21.15.72:0/3749287534 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f34145e80 con 0x7f5f340b3cd0 2024-03-29T11:37:30.146 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.145+0000 7f5f327fc640 10 monclient: _finish_auth 0 2024-03-29T11:37:30.146 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.145+0000 7f5f327fc640 10 monclient: _check_auth_tickets 2024-03-29T11:37:30.146 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.145+0000 7f5f327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:30.147 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.145+0000 7f5f327fc640 1 -- 172.21.15.72:0/3749287534 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f5f2c0019e0 con 0x7f5f340b3cd0 2024-03-29T11:37:30.147 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.145+0000 7f5f327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:30.147 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.145+0000 7f5f327fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:30.147 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.145+0000 7f5f327fc640 10 monclient: dump: 2024-03-29T11:37:30.147 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:37:30.147 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:30.147 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:30.147 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:30.147 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:30.147 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:30.147 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:37:30.147 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:37:30.147 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:37:30.148 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:30.148 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.145+0000 7f5f327fc640 1 -- 172.21.15.72:0/3749287534 <== mon.0 v1:172.21.15.72:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f2c001b40 con 0x7f5f340b3cd0 2024-03-29T11:37:30.148 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.145+0000 7f5f327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:30.148 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.145+0000 7f5f327fc640 1 -- 172.21.15.72:0/3749287534 <== mon.0 v1:172.21.15.72:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f5f2c001f50 con 0x7f5f340b3cd0 2024-03-29T11:37:30.148 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.145+0000 7f5f327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:30.148 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.145+0000 7f5f327fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:30.148 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.145+0000 7f5f327fc640 10 monclient: dump: 2024-03-29T11:37:30.148 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:37:30.148 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:30.149 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:30.149 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:30.149 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:30.149 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:30.149 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:37:30.149 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:37:30.149 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:37:30.149 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:30.150 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.148+0000 7f5f38afe640 5 monclient: authenticate success, global_id 4314 2024-03-29T11:37:30.150 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.148+0000 7f5f38afe640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:30.150 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.148+0000 7f5f38afe640 4 set_mon_vals no callback set 2024-03-29T11:37:30.150 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.149+0000 7f5f38afe640 10 monclient: shutdown 2024-03-29T11:37:30.150 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.149+0000 7f5f38afe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:30.150 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.149+0000 7f5f38afe640 1 -- 172.21.15.72:0/3749287534 >> v1:172.21.15.72:6789/0 conn(0x7f5f340b3cd0 legacy=0x7f5f3414d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:30.151 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.151+0000 7f5f38afe640 1 -- 172.21.15.72:0/3749287534 shutdown_connections 2024-03-29T11:37:30.151 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.151+0000 7f5f38afe640 1 -- 172.21.15.72:0/3749287534 >> 172.21.15.72:0/3749287534 conn(0x7f5f3406d070 msgr2=0x7f5f34066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:30.154 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.153+0000 7f5f38afe640 1 -- 172.21.15.72:0/3749287534 shutdown_connections 2024-03-29T11:37:30.154 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.155+0000 7f5f38afe640 1 -- 172.21.15.72:0/3749287534 wait complete. 2024-03-29T11:37:30.155 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.156+0000 7f5f38afe640 10 monclient: build_initial_monmap 2024-03-29T11:37:30.155 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.156+0000 7f5f38afe640 10 monclient: monmap: 2024-03-29T11:37:30.155 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:37:30.156 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:30.156 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:37:30.156 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:37:30.156 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:30.156 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:30.156 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:37:30.156 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:37:30.156 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:37:30.156 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:30.158 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.156+0000 7f5f38afe640 1 librados: starting msgr at 2024-03-29T11:37:30.158 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.156+0000 7f5f38afe640 1 librados: starting objecter 2024-03-29T11:37:30.158 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.156+0000 7f5f38afe640 1 Processor -- start 2024-03-29T11:37:30.158 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.157+0000 7f5f38afe640 1 -- start start 2024-03-29T11:37:30.158 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.157+0000 7f5f38afe640 1 librados: setting wanted keys 2024-03-29T11:37:30.158 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.157+0000 7f5f38afe640 1 librados: calling monclient init 2024-03-29T11:37:30.158 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.157+0000 7f5f38afe640 10 monclient: init 2024-03-29T11:37:30.158 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.157+0000 7f5f38afe640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:30.159 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.157+0000 7f5f38afe640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:37:30.159 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.157+0000 7f5f38afe640 10 monclient(hunting): picked mon.noname-c con 0x7f5f340a9ce0 addr v1:172.21.15.72:6790/0 2024-03-29T11:37:30.159 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.157+0000 7f5f38afe640 10 monclient(hunting): picked mon.noname-a con 0x7f5f340b3cd0 addr v1:172.21.15.72:6789/0 2024-03-29T11:37:30.159 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.157+0000 7f5f38afe640 10 monclient(hunting): picked mon.noname-b con 0x7f5f340b4820 addr v1:172.21.15.89:6789/0 2024-03-29T11:37:30.159 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.157+0000 7f5f38afe640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f3413af60 con 0x7f5f340b3cd0 2024-03-29T11:37:30.159 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.157+0000 7f5f38afe640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f341d7470 con 0x7f5f340b4820 2024-03-29T11:37:30.159 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.158+0000 7f5f38afe640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f341361b0 con 0x7f5f340a9ce0 2024-03-29T11:37:30.159 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.158+0000 7f5f38afe640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:30.159 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.158+0000 7f5f38afe640 10 monclient(hunting): authenticate will time out at 1009.446014s 2024-03-29T11:37:30.159 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.158+0000 7f5f33fff640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f5f340b4820 0x7f5f341d44a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:37830/0 (socket says 172.21.15.72:37830) 2024-03-29T11:37:30.159 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.158+0000 7f5f33fff640 1 -- 172.21.15.72:0/3904625747 learned_addr learned my addr 172.21.15.72:0/3904625747 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:37:30.165 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.161+0000 7f5f2bfff640 10 client.?.objecter ms_handle_connect 0x7f5f340b4820 2024-03-29T11:37:30.165 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.161+0000 7f5f2bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:30.165 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.161+0000 7f5f2bfff640 10 client.?.objecter ms_handle_connect 0x7f5f340b3cd0 2024-03-29T11:37:30.165 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.161+0000 7f5f2bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:30.165 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.161+0000 7f5f2bfff640 1 -- 172.21.15.72:0/3904625747 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 12846073 0 0) 0x7f5f3413af60 con 0x7f5f340b3cd0 2024-03-29T11:37:30.165 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.161+0000 7f5f2bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:37:30.165 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.161+0000 7f5f2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:30.165 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.161+0000 7f5f2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:30.165 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.161+0000 7f5f2bfff640 10 monclient(hunting): my global_id is 4317 2024-03-29T11:37:30.165 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.161+0000 7f5f2bfff640 1 -- 172.21.15.72:0/3904625747 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f14004eb0 con 0x7f5f340b3cd0 2024-03-29T11:37:30.165 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.161+0000 7f5f2bfff640 1 -- 172.21.15.72:0/3904625747 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 234452008 0 0) 0x7f5f341d7470 con 0x7f5f340b4820 2024-03-29T11:37:30.165 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.161+0000 7f5f2bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:37:30.165 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.161+0000 7f5f2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:30.166 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.161+0000 7f5f2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:30.166 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.161+0000 7f5f2bfff640 10 monclient(hunting): my global_id is 4312 2024-03-29T11:37:30.166 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.161+0000 7f5f2bfff640 1 -- 172.21.15.72:0/3904625747 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f3413af60 con 0x7f5f340b4820 2024-03-29T11:37:30.166 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.161+0000 7f5f2bfff640 10 client.?.objecter ms_handle_connect 0x7f5f340a9ce0 2024-03-29T11:37:30.166 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.161+0000 7f5f2bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:30.166 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.162+0000 7f5f2bfff640 1 -- 172.21.15.72:0/3904625747 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3035206765 0 0) 0x7f5f341361b0 con 0x7f5f340a9ce0 2024-03-29T11:37:30.166 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.162+0000 7f5f2bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:37:30.166 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.162+0000 7f5f2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:30.166 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.162+0000 7f5f2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:30.166 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.162+0000 7f5f2bfff640 10 monclient(hunting): my global_id is 4313 2024-03-29T11:37:30.166 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.162+0000 7f5f2bfff640 1 -- 172.21.15.72:0/3904625747 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f341d7470 con 0x7f5f340a9ce0 2024-03-29T11:37:30.166 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.162+0000 7f5f2bfff640 1 -- 172.21.15.72:0/3904625747 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3469068498 0 0) 0x7f5f14004eb0 con 0x7f5f340b3cd0 2024-03-29T11:37:30.166 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.162+0000 7f5f2bfff640 1 -- 172.21.15.72:0/3904625747 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f341361b0 con 0x7f5f340b3cd0 2024-03-29T11:37:30.166 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.162+0000 7f5f2bfff640 1 -- 172.21.15.72:0/3904625747 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f5f1c001950 con 0x7f5f340b3cd0 2024-03-29T11:37:30.167 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.162+0000 7f5f2bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:30.167 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.162+0000 7f5f2bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:30.167 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.162+0000 7f5f2bfff640 10 monclient(hunting): dump: 2024-03-29T11:37:30.167 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:37:30.167 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:30.167 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:30.167 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:30.167 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:30.167 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:30.167 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:37:30.167 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:37:30.167 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:37:30.167 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:30.167 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.162+0000 7f5f2bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:30.167 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.162+0000 7f5f2bfff640 1 -- 172.21.15.72:0/3904625747 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2670697471 0 0) 0x7f5f3413af60 con 0x7f5f340b4820 2024-03-29T11:37:30.168 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.162+0000 7f5f2bfff640 1 -- 172.21.15.72:0/3904625747 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f14004eb0 con 0x7f5f340b4820 2024-03-29T11:37:30.168 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.162+0000 7f5f2bfff640 1 -- 172.21.15.72:0/3904625747 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f5f2c001e50 con 0x7f5f340b4820 2024-03-29T11:37:30.168 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.162+0000 7f5f2bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:30.168 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.162+0000 7f5f2bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:30.168 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.162+0000 7f5f2bfff640 10 monclient(hunting): dump: 2024-03-29T11:37:30.168 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:37:30.168 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:30.168 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:30.168 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:30.168 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:30.168 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:30.168 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:37:30.168 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:37:30.168 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:37:30.168 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:30.168 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.162+0000 7f5f2bfff640 1 -- 172.21.15.72:0/3904625747 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 74692853 0 0) 0x7f5f341d7470 con 0x7f5f340a9ce0 2024-03-29T11:37:30.168 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.162+0000 7f5f2bfff640 1 -- 172.21.15.72:0/3904625747 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f3413af60 con 0x7f5f340a9ce0 2024-03-29T11:37:30.169 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.162+0000 7f5f2bfff640 1 -- 172.21.15.72:0/3904625747 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f5f240016f0 con 0x7f5f340a9ce0 2024-03-29T11:37:30.169 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.162+0000 7f5f2bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:30.169 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.162+0000 7f5f2bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:30.169 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.162+0000 7f5f2bfff640 10 monclient(hunting): dump: 2024-03-29T11:37:30.169 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:37:30.169 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:30.169 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:30.169 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:30.169 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:30.169 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:30.169 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:37:30.169 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:37:30.169 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:37:30.169 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:30.169 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.163+0000 7f5f2bfff640 1 -- 172.21.15.72:0/3904625747 <== mon.0 v1:172.21.15.72:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 910853975 0 0) 0x7f5f341361b0 con 0x7f5f340b3cd0 2024-03-29T11:37:30.169 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.163+0000 7f5f2bfff640 1 -- 172.21.15.72:0/3904625747 >> v1:172.21.15.72:6790/0 conn(0x7f5f340a9ce0 legacy=0x7f5f3413a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:30.170 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.163+0000 7f5f2bfff640 1 -- 172.21.15.72:0/3904625747 >> v1:172.21.15.89:6789/0 conn(0x7f5f340b4820 legacy=0x7f5f341d44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:30.170 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.163+0000 7f5f2bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:37:30.170 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.163+0000 7f5f2bfff640 1 monclient: found mon.a 2024-03-29T11:37:30.170 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.163+0000 7f5f2bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:30.170 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.163+0000 7f5f2bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:37:30.170 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.163+0000 7f5f2bfff640 1 -- 172.21.15.72:0/3904625747 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f341d3ed0 con 0x7f5f340b3cd0 2024-03-29T11:37:30.170 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.163+0000 7f5f2bfff640 10 monclient: _finish_auth 0 2024-03-29T11:37:30.170 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.163+0000 7f5f2bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:37:30.170 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.163+0000 7f5f2bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:30.170 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.163+0000 7f5f38afe640 5 monclient: authenticate success, global_id 4317 2024-03-29T11:37:30.170 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.163+0000 7f5f38afe640 10 monclient: _renew_subs 2024-03-29T11:37:30.170 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.163+0000 7f5f38afe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:37:30.170 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.163+0000 7f5f38afe640 1 -- 172.21.15.72:0/3904625747 --> v1:172.21.15.72:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f341d8080 con 0x7f5f340b3cd0 2024-03-29T11:37:30.170 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.163+0000 7f5f2bfff640 1 -- 172.21.15.72:0/3904625747 <== mon.0 v1:172.21.15.72:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f1c001d60 con 0x7f5f340b3cd0 2024-03-29T11:37:30.170 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.163+0000 7f5f2bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:30.171 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.163+0000 7f5f38afe640 10 client.4317.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:30.171 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.163+0000 7f5f31ffb640 4 set_mon_vals no callback set 2024-03-29T11:37:30.171 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.163+0000 7f5f38afe640 10 monclient: _renew_subs 2024-03-29T11:37:30.171 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.163+0000 7f5f38afe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:37:30.171 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.163+0000 7f5f38afe640 1 -- 172.21.15.72:0/3904625747 --> v1:172.21.15.72:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f341d8330 con 0x7f5f340b3cd0 2024-03-29T11:37:30.171 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.163+0000 7f5f38afe640 1 librados: init done 2024-03-29T11:37:30.171 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.163+0000 7f5f2bfff640 1 -- 172.21.15.72:0/3904625747 <== mon.0 v1:172.21.15.72:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f5f1c004980 con 0x7f5f340b3cd0 2024-03-29T11:37:30.171 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.163+0000 7f5f2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:30.171 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.163+0000 7f5f2bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:30.171 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.163+0000 7f5f2bfff640 10 monclient: dump: 2024-03-29T11:37:30.171 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:37:30.171 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:30.171 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:30.171 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:30.171 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:30.171 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:30.172 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:37:30.172 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:37:30.172 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:37:30.172 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:30.172 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.164+0000 7f5f38afe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:30.172 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.164+0000 7f5f38afe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:30.172 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.164+0000 7f5f38afe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:37:30.172 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.164+0000 7f5f38afe640 1 -- 172.21.15.72:0/3904625747 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f341d7620 con 0x7f5f340b3cd0 2024-03-29T11:37:30.173 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.165+0000 7f5f2bfff640 1 -- 172.21.15.72:0/3904625747 <== mon.0 v1:172.21.15.72:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f5f1c01a9d0 con 0x7f5f340b3cd0 2024-03-29T11:37:30.173 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.165+0000 7f5f337fe640 10 monclient: get_auth_request con 0x7f5f1406e1d0 auth_method 0 2024-03-29T11:37:30.173 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.166+0000 7f5f2bfff640 1 -- 172.21.15.72:0/3904625747 <== mon.0 v1:172.21.15.72:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f5f1c0849a0 con 0x7f5f340b3cd0 2024-03-29T11:37:30.173 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.166+0000 7f5f2bfff640 10 client.4317.objecter ms_dispatch 0x7f5f34000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:30.173 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.166+0000 7f5f2bfff640 3 client.4317.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:30.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.166+0000 7f5f2bfff640 3 client.4317.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:30.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.166+0000 7f5f2bfff640 20 client.4317.objecter dump_active .. 0 homeless 2024-03-29T11:37:30.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.166+0000 7f5f2bfff640 10 client.4317.objecter ms_handle_connect 0x7f5f1406e1d0 2024-03-29T11:37:30.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.173+0000 7f5f2bfff640 1 -- 172.21.15.72:0/3904625747 <== mon.0 v1:172.21.15.72:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5f341d7620 con 0x7f5f340b3cd0 2024-03-29T11:37:30.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.173+0000 7f5f2bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:30.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.173+0000 7f5f2bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:30.494 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.495+0000 7f5f38afe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:30.495 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.495+0000 7f5f38afe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:30.495 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.495+0000 7f5f38afe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:37:30.495 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.495+0000 7f5f38afe640 1 -- 172.21.15.72:0/3904625747 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f341d7620 con 0x7f5f340b3cd0 2024-03-29T11:37:30.496 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.496+0000 7f5f2bfff640 1 -- 172.21.15.72:0/3904625747 <== mon.0 v1:172.21.15.72:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f5f341d7620 con 0x7f5f340b3cd0 2024-03-29T11:37:30.496 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.496+0000 7f5f2bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:30.496 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.496+0000 7f5f2bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:30.497 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:37:30.497 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:30.499 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.500+0000 7f5f38afe640 10 librados: watch_flush enter 2024-03-29T11:37:30.499 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.500+0000 7f5f38afe640 10 librados: watch_flush exit 2024-03-29T11:37:30.500 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.500+0000 7f5f38afe640 20 client.4317.objecter shutdown clearing up homeless session... 2024-03-29T11:37:30.500 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.500+0000 7f5f38afe640 10 client.4317.objecter successfully canceled tick 2024-03-29T11:37:30.501 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.500+0000 7f5f38afe640 1 -- 172.21.15.72:0/3904625747 >> v1:172.21.15.72:6816/1578866102 conn(0x7f5f1406e1d0 legacy=0x7f5f14070690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:30.501 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.500+0000 7f5f38afe640 10 monclient: shutdown 2024-03-29T11:37:30.501 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.500+0000 7f5f38afe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:30.501 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.500+0000 7f5f38afe640 1 -- 172.21.15.72:0/3904625747 >> v1:172.21.15.72:6789/0 conn(0x7f5f340b3cd0 legacy=0x7f5f341d7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:30.501 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.500+0000 7f5f38afe640 1 -- 172.21.15.72:0/3904625747 shutdown_connections 2024-03-29T11:37:30.501 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.500+0000 7f5f38afe640 1 -- 172.21.15.72:0/3904625747 >> 172.21.15.72:0/3904625747 conn(0x7f5f3406d070 msgr2=0x7f5f340b8220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:30.502 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.501+0000 7f5f38afe640 1 -- 172.21.15.72:0/3904625747 shutdown_connections 2024-03-29T11:37:30.502 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.501+0000 7f5f38afe640 1 -- 172.21.15.72:0/3904625747 wait complete. 2024-03-29T11:37:30.502 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:30.501+0000 7f5f38afe640 1 librados: shutdown 2024-03-29T11:37:30.515 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:37.323 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:37.471 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.471+0000 7f144e60d640 10 monclient: get_monmap_and_config 2024-03-29T11:37:37.471 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.471+0000 7f144e60d640 10 monclient: build_initial_monmap 2024-03-29T11:37:37.471 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.471+0000 7f144e60d640 10 monclient: monmap: 2024-03-29T11:37:37.471 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:37:37.471 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:37.471 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:37:37.472572+0000 2024-03-29T11:37:37.472 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:37:37.472572+0000 2024-03-29T11:37:37.472 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:37.472 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:37.472 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:37:37.472 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:37:37.472 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:37:37.472 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:37.473 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.473+0000 7f144e60d640 1 Processor -- start 2024-03-29T11:37:37.473 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.473+0000 7f144e60d640 1 -- start start 2024-03-29T11:37:37.473 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.473+0000 7f144e60d640 10 monclient: init 2024-03-29T11:37:37.477 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.474+0000 7f144e60d640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:37.478 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.474+0000 7f144e60d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:37:37.478 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.474+0000 7f144e60d640 10 monclient(hunting): picked mon.noname-b con 0x7f14481457f0 addr v1:172.21.15.72:6790/0 2024-03-29T11:37:37.478 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.474+0000 7f144e60d640 10 monclient(hunting): picked mon.noname-a con 0x7f144813f650 addr v1:172.21.15.72:6789/0 2024-03-29T11:37:37.478 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.474+0000 7f144e60d640 10 monclient(hunting): picked mon.noname-c con 0x7f14480a9d70 addr v1:172.21.15.89:6789/0 2024-03-29T11:37:37.478 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.474+0000 7f144e60d640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14480aaef0 con 0x7f144813f650 2024-03-29T11:37:37.478 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.474+0000 7f144e60d640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1448145bd0 con 0x7f14480a9d70 2024-03-29T11:37:37.478 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.474+0000 7f144e60d640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1448145eb0 con 0x7f14481457f0 2024-03-29T11:37:37.478 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.474+0000 7f144e60d640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:37.478 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.474+0000 7f144e60d640 10 monclient(hunting): authenticate will time out at 1016.762781s 2024-03-29T11:37:37.480 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.480+0000 7f144cbaa640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f14480a9d70 0x7f144814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:60969/0 (socket says 172.21.15.72:60969) 2024-03-29T11:37:37.480 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.480+0000 7f144cbaa640 1 -- 172.21.15.72:0/849973801 learned_addr learned my addr 172.21.15.72:0/849973801 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:37:37.482 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.483+0000 7f1446ffd640 1 -- 172.21.15.72:0/849973801 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4231188729 0 0) 0x7f1448145eb0 con 0x7f14481457f0 2024-03-29T11:37:37.482 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.483+0000 7f1446ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:37:37.482 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.483+0000 7f1446ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:37.483 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.483+0000 7f1446ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:37.483 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.483+0000 7f1446ffd640 10 monclient(hunting): my global_id is 4316 2024-03-29T11:37:37.483 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.483+0000 7f1446ffd640 1 -- 172.21.15.72:0/849973801 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1430003520 con 0x7f14481457f0 2024-03-29T11:37:37.483 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.483+0000 7f1446ffd640 1 -- 172.21.15.72:0/849973801 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 165633028 0 0) 0x7f14480aaef0 con 0x7f144813f650 2024-03-29T11:37:37.483 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.483+0000 7f1446ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:37:37.483 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.483+0000 7f1446ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:37.483 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.483+0000 7f1446ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:37.483 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.483+0000 7f1446ffd640 10 monclient(hunting): my global_id is 4320 2024-03-29T11:37:37.483 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.483+0000 7f1446ffd640 1 -- 172.21.15.72:0/849973801 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1448145eb0 con 0x7f144813f650 2024-03-29T11:37:37.484 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.484+0000 7f1446ffd640 1 -- 172.21.15.72:0/849973801 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 44598713 0 0) 0x7f1448145eb0 con 0x7f144813f650 2024-03-29T11:37:37.484 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.484+0000 7f1446ffd640 1 -- 172.21.15.72:0/849973801 >> v1:172.21.15.72:6790/0 conn(0x7f14481457f0 legacy=0x7f144813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:37.484 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.484+0000 7f1446ffd640 1 -- 172.21.15.72:0/849973801 >> v1:172.21.15.89:6789/0 conn(0x7f14480a9d70 legacy=0x7f144814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:37.484 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.484+0000 7f1446ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:37:37.484 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.484+0000 7f1446ffd640 1 monclient: found mon.noname-a 2024-03-29T11:37:37.484 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.484+0000 7f1446ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:37.484 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.484+0000 7f1446ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.72:6789/0 2024-03-29T11:37:37.484 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.484+0000 7f1446ffd640 1 -- 172.21.15.72:0/849973801 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1448146340 con 0x7f144813f650 2024-03-29T11:37:37.484 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.484+0000 7f1446ffd640 10 monclient: _finish_auth 0 2024-03-29T11:37:37.485 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.484+0000 7f1446ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:37:37.485 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.484+0000 7f1446ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:37.485 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.484+0000 7f1446ffd640 1 -- 172.21.15.72:0/849973801 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f143c001520 con 0x7f144813f650 2024-03-29T11:37:37.485 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.484+0000 7f1446ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:37.485 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.484+0000 7f1446ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:37.485 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.484+0000 7f1446ffd640 10 monclient: dump: 2024-03-29T11:37:37.485 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:37:37.485 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:37.485 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:37.485 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:37.485 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:37.485 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:37.485 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:37:37.485 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:37:37.485 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:37:37.486 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:37.486 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.484+0000 7f144e60d640 5 monclient: authenticate success, global_id 4320 2024-03-29T11:37:37.486 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.484+0000 7f144e60d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:37.486 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.485+0000 7f1446ffd640 1 -- 172.21.15.72:0/849973801 <== mon.0 v1:172.21.15.72:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f143c001680 con 0x7f144813f650 2024-03-29T11:37:37.486 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.485+0000 7f1446ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:37.487 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.485+0000 7f1446ffd640 1 -- 172.21.15.72:0/849973801 <== mon.0 v1:172.21.15.72:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f143c001a10 con 0x7f144813f650 2024-03-29T11:37:37.487 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.485+0000 7f144e60d640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:37.487 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.485+0000 7f144e60d640 4 set_mon_vals no callback set 2024-03-29T11:37:37.487 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.485+0000 7f1446ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:37.487 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.485+0000 7f1446ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:37.487 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.485+0000 7f1446ffd640 10 monclient: dump: 2024-03-29T11:37:37.487 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:37:37.487 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:37.487 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:37.487 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:37.487 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:37.487 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:37.488 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:37:37.488 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:37:37.488 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:37:37.488 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:37.489 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.489+0000 7f144e60d640 10 monclient: shutdown 2024-03-29T11:37:37.489 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.489+0000 7f144e60d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:37.489 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.489+0000 7f144e60d640 1 -- 172.21.15.72:0/849973801 >> v1:172.21.15.72:6789/0 conn(0x7f144813f650 legacy=0x7f14480b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:37.489 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.489+0000 7f144e60d640 1 -- 172.21.15.72:0/849973801 shutdown_connections 2024-03-29T11:37:37.489 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.489+0000 7f144e60d640 1 -- 172.21.15.72:0/849973801 >> 172.21.15.72:0/849973801 conn(0x7f1448066a00 msgr2=0x7f1448066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:37.492 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.492+0000 7f144e60d640 1 -- 172.21.15.72:0/849973801 shutdown_connections 2024-03-29T11:37:37.492 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.492+0000 7f144e60d640 1 -- 172.21.15.72:0/849973801 wait complete. 2024-03-29T11:37:37.500 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.493+0000 7f144e60d640 10 monclient: build_initial_monmap 2024-03-29T11:37:37.500 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.493+0000 7f144e60d640 10 monclient: monmap: 2024-03-29T11:37:37.500 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:37:37.500 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:37.500 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:37:37.500 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:37:37.500 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:37.500 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:37.501 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:37:37.501 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:37:37.501 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:37:37.501 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:37.501 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.493+0000 7f144e60d640 1 librados: starting msgr at 2024-03-29T11:37:37.501 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.493+0000 7f144e60d640 1 librados: starting objecter 2024-03-29T11:37:37.501 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.493+0000 7f144e60d640 1 Processor -- start 2024-03-29T11:37:37.501 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.494+0000 7f144e60d640 1 -- start start 2024-03-29T11:37:37.501 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.494+0000 7f144e60d640 1 librados: setting wanted keys 2024-03-29T11:37:37.501 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.494+0000 7f144e60d640 1 librados: calling monclient init 2024-03-29T11:37:37.501 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.494+0000 7f144e60d640 10 monclient: init 2024-03-29T11:37:37.501 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.494+0000 7f144e60d640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:37.501 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.494+0000 7f144e60d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:37:37.501 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.494+0000 7f144e60d640 10 monclient(hunting): picked mon.noname-a con 0x7f14480a9d70 addr v1:172.21.15.72:6789/0 2024-03-29T11:37:37.501 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.494+0000 7f144e60d640 10 monclient(hunting): picked mon.noname-c con 0x7f144813f650 addr v1:172.21.15.72:6790/0 2024-03-29T11:37:37.501 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.494+0000 7f144e60d640 10 monclient(hunting): picked mon.noname-b con 0x7f14481457f0 addr v1:172.21.15.89:6789/0 2024-03-29T11:37:37.502 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.494+0000 7f144e60d640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1448137be0 con 0x7f14480a9d70 2024-03-29T11:37:37.502 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.494+0000 7f144e60d640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f144806e320 con 0x7f14481457f0 2024-03-29T11:37:37.502 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.494+0000 7f144e60d640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f144806ebe0 con 0x7f144813f650 2024-03-29T11:37:37.502 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.494+0000 7f144e60d640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:37.502 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.494+0000 7f144e60d640 10 monclient(hunting): authenticate will time out at 1016.782800s 2024-03-29T11:37:37.502 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.498+0000 7f144cbaa640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f14481457f0 0x7f14481374d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:32833/0 (socket says 172.21.15.72:32833) 2024-03-29T11:37:37.502 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.498+0000 7f144cbaa640 1 -- 172.21.15.72:0/2016643627 learned_addr learned my addr 172.21.15.72:0/2016643627 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:37:37.502 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.499+0000 7f1444ff9640 10 client.?.objecter ms_handle_connect 0x7f14481457f0 2024-03-29T11:37:37.502 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.499+0000 7f1444ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:37.502 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.499+0000 7f1444ff9640 1 -- 172.21.15.72:0/2016643627 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1066769296 0 0) 0x7f144806e320 con 0x7f14481457f0 2024-03-29T11:37:37.503 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.500+0000 7f1444ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:37:37.503 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.500+0000 7f1444ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:37.503 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.500+0000 7f1444ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:37.503 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.500+0000 7f1444ff9640 10 monclient(hunting): my global_id is 4315 2024-03-29T11:37:37.503 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.500+0000 7f1444ff9640 1 -- 172.21.15.72:0/2016643627 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1434004eb0 con 0x7f14481457f0 2024-03-29T11:37:37.503 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.500+0000 7f1444ff9640 10 client.?.objecter ms_handle_connect 0x7f144813f650 2024-03-29T11:37:37.503 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.500+0000 7f1444ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:37.504 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.503+0000 7f1444ff9640 10 client.?.objecter ms_handle_connect 0x7f14480a9d70 2024-03-29T11:37:37.505 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.503+0000 7f1444ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:37.505 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.503+0000 7f1444ff9640 1 -- 172.21.15.72:0/2016643627 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2741144654 0 0) 0x7f1434004eb0 con 0x7f14481457f0 2024-03-29T11:37:37.505 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.503+0000 7f1444ff9640 1 -- 172.21.15.72:0/2016643627 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f144806e320 con 0x7f14481457f0 2024-03-29T11:37:37.505 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.503+0000 7f1444ff9640 1 -- 172.21.15.72:0/2016643627 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3377746533 0 0) 0x7f144806ebe0 con 0x7f144813f650 2024-03-29T11:37:37.505 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.503+0000 7f1444ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:37:37.507 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.503+0000 7f1444ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:37.507 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.503+0000 7f1444ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:37.507 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.503+0000 7f1444ff9640 10 monclient(hunting): my global_id is 4319 2024-03-29T11:37:37.507 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.503+0000 7f1444ff9640 1 -- 172.21.15.72:0/2016643627 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1434004eb0 con 0x7f144813f650 2024-03-29T11:37:37.507 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.503+0000 7f1444ff9640 1 -- 172.21.15.72:0/2016643627 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f1440001c30 con 0x7f14481457f0 2024-03-29T11:37:37.507 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.503+0000 7f1444ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:37.507 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.503+0000 7f1444ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:37.511 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.503+0000 7f1444ff9640 10 monclient(hunting): dump: 2024-03-29T11:37:37.511 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:37:37.511 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:37.511 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:37.511 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:37.511 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:37.511 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:37.512 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:37:37.512 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:37:37.512 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:37:37.512 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:37.512 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.503+0000 7f1444ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:37.512 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.503+0000 7f1444ff9640 1 -- 172.21.15.72:0/2016643627 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 667835031 0 0) 0x7f1434004eb0 con 0x7f144813f650 2024-03-29T11:37:37.512 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.503+0000 7f1444ff9640 1 -- 172.21.15.72:0/2016643627 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f144806ebe0 con 0x7f144813f650 2024-03-29T11:37:37.512 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.503+0000 7f1444ff9640 1 -- 172.21.15.72:0/2016643627 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f143c001240 con 0x7f144813f650 2024-03-29T11:37:37.512 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.503+0000 7f1444ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:37.513 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.503+0000 7f1444ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:37.513 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.503+0000 7f1444ff9640 10 monclient(hunting): dump: 2024-03-29T11:37:37.513 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:37:37.513 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:37.513 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:37.513 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:37.513 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:37.513 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:37.513 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:37:37.513 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:37:37.513 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:37:37.513 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:37.513 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.504+0000 7f1444ff9640 1 -- 172.21.15.72:0/2016643627 <== mon.2 v1:172.21.15.72:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1452861827 0 0) 0x7f144806ebe0 con 0x7f144813f650 2024-03-29T11:37:37.513 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.504+0000 7f1444ff9640 1 -- 172.21.15.72:0/2016643627 >> v1:172.21.15.89:6789/0 conn(0x7f14481457f0 legacy=0x7f14481374d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:37.513 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.504+0000 7f1444ff9640 1 -- 172.21.15.72:0/2016643627 >> v1:172.21.15.72:6789/0 conn(0x7f14480a9d70 legacy=0x7f144806f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:37.514 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.504+0000 7f1444ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:37:37.514 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.504+0000 7f1444ff9640 1 monclient: found mon.c 2024-03-29T11:37:37.514 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.504+0000 7f1444ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:37.514 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.504+0000 7f1444ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:37:37.514 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.504+0000 7f1444ff9640 1 -- 172.21.15.72:0/2016643627 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f144806eec0 con 0x7f144813f650 2024-03-29T11:37:37.514 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.504+0000 7f1444ff9640 10 monclient: _finish_auth 0 2024-03-29T11:37:37.514 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.504+0000 7f1444ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:37:37.514 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.504+0000 7f1444ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:37.514 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.504+0000 7f1444ff9640 1 -- 172.21.15.72:0/2016643627 <== mon.2 v1:172.21.15.72:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f143c0038a0 con 0x7f144813f650 2024-03-29T11:37:37.514 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.504+0000 7f1444ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:37.514 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.504+0000 7f1445ffb640 4 set_mon_vals no callback set 2024-03-29T11:37:37.514 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.504+0000 7f1444ff9640 1 -- 172.21.15.72:0/2016643627 <== mon.2 v1:172.21.15.72:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f143c003bb0 con 0x7f144813f650 2024-03-29T11:37:37.514 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.504+0000 7f1444ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:37.514 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.504+0000 7f1444ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:37.514 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.504+0000 7f1444ff9640 10 monclient: dump: 2024-03-29T11:37:37.515 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:37:37.515 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:37.515 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:37.515 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:37.515 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:37.515 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:37.515 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:37:37.515 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:37:37.515 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:37:37.515 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:37.515 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.505+0000 7f144e60d640 5 monclient: authenticate success, global_id 4319 2024-03-29T11:37:37.515 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.505+0000 7f144e60d640 10 monclient: _renew_subs 2024-03-29T11:37:37.515 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.505+0000 7f144e60d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:37:37.515 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.505+0000 7f144e60d640 1 -- 172.21.15.72:0/2016643627 --> v1:172.21.15.72:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f144806f1a0 con 0x7f144813f650 2024-03-29T11:37:37.515 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.505+0000 7f144e60d640 10 client.4319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:37.515 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.505+0000 7f144e60d640 10 monclient: _renew_subs 2024-03-29T11:37:37.516 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.505+0000 7f144e60d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:37:37.516 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.505+0000 7f144e60d640 1 -- 172.21.15.72:0/2016643627 --> v1:172.21.15.72:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f144806e920 con 0x7f144813f650 2024-03-29T11:37:37.516 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.505+0000 7f144e60d640 1 librados: init done 2024-03-29T11:37:37.516 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.507+0000 7f144e60d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:37.516 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.507+0000 7f144e60d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:37.516 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.507+0000 7f144e60d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:37:37.516 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.507+0000 7f144e60d640 1 -- 172.21.15.72:0/2016643627 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1448075f10 con 0x7f144813f650 2024-03-29T11:37:37.516 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.510+0000 7f1444ff9640 1 -- 172.21.15.72:0/2016643627 <== mon.2 v1:172.21.15.72:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f143c019b80 con 0x7f144813f650 2024-03-29T11:37:37.516 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.510+0000 7f1447fff640 10 monclient: get_auth_request con 0x7f143406de50 auth_method 0 2024-03-29T11:37:37.516 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.511+0000 7f1444ff9640 10 client.4319.objecter ms_handle_connect 0x7f143406de50 2024-03-29T11:37:37.516 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.511+0000 7f1444ff9640 1 -- 172.21.15.72:0/2016643627 <== mon.2 v1:172.21.15.72:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f143c084940 con 0x7f144813f650 2024-03-29T11:37:37.516 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.511+0000 7f1444ff9640 10 client.4319.objecter ms_dispatch 0x7f1448000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:37.516 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.511+0000 7f1444ff9640 3 client.4319.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:37.516 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.511+0000 7f1444ff9640 3 client.4319.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:37.516 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.511+0000 7f1444ff9640 20 client.4319.objecter dump_active .. 0 homeless 2024-03-29T11:37:37.518 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.518+0000 7f1444ff9640 1 -- 172.21.15.72:0/2016643627 <== mon.2 v1:172.21.15.72:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f143c04f1f0 con 0x7f144813f650 2024-03-29T11:37:37.518 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.518+0000 7f1444ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:37.518 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.518+0000 7f1444ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:37.851 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.851+0000 7f144e60d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:37.852 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.852+0000 7f144e60d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:37.852 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.852+0000 7f144e60d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:37:37.852 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.852+0000 7f144e60d640 1 -- 172.21.15.72:0/2016643627 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14480b3680 con 0x7f144813f650 2024-03-29T11:37:37.854 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.854+0000 7f1444ff9640 1 -- 172.21.15.72:0/2016643627 <== mon.2 v1:172.21.15.72:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f143c051120 con 0x7f144813f650 2024-03-29T11:37:37.854 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.854+0000 7f1444ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:37.854 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:37:37.854 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:37.855 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.854+0000 7f1444ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:37.859 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.859+0000 7f144e60d640 10 librados: watch_flush enter 2024-03-29T11:37:37.859 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.859+0000 7f144e60d640 10 librados: watch_flush exit 2024-03-29T11:37:37.859 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.859+0000 7f144e60d640 20 client.4319.objecter shutdown clearing up homeless session... 2024-03-29T11:37:37.859 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.859+0000 7f144e60d640 10 client.4319.objecter successfully canceled tick 2024-03-29T11:37:37.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.859+0000 7f144e60d640 1 -- 172.21.15.72:0/2016643627 >> v1:172.21.15.72:6816/1578866102 conn(0x7f143406de50 legacy=0x7f1434070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:37.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.859+0000 7f144e60d640 10 monclient: shutdown 2024-03-29T11:37:37.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.859+0000 7f144e60d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:37.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.859+0000 7f144e60d640 1 -- 172.21.15.72:0/2016643627 >> v1:172.21.15.72:6790/0 conn(0x7f144813f650 legacy=0x7f1448136dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:37.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.859+0000 7f144e60d640 1 -- 172.21.15.72:0/2016643627 shutdown_connections 2024-03-29T11:37:37.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.859+0000 7f144e60d640 1 -- 172.21.15.72:0/2016643627 >> 172.21.15.72:0/2016643627 conn(0x7f1448066a00 msgr2=0x7f14480b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:37.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.860+0000 7f144e60d640 1 -- 172.21.15.72:0/2016643627 shutdown_connections 2024-03-29T11:37:37.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.860+0000 7f144e60d640 1 -- 172.21.15.72:0/2016643627 wait complete. 2024-03-29T11:37:37.861 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:37.861+0000 7f144e60d640 1 librados: shutdown 2024-03-29T11:37:37.885 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:44.691 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:44.846 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.845+0000 7f9c038a8640 10 monclient: get_monmap_and_config 2024-03-29T11:37:44.846 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.845+0000 7f9c038a8640 10 monclient: build_initial_monmap 2024-03-29T11:37:44.846 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.845+0000 7f9c038a8640 10 monclient: monmap: 2024-03-29T11:37:44.847 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:37:44.847 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:44.847 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:37:44.846708+0000 2024-03-29T11:37:44.847 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:37:44.846708+0000 2024-03-29T11:37:44.847 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:44.847 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:44.847 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:37:44.847 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:37:44.847 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:37:44.847 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:44.848 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.848+0000 7f9c038a8640 1 Processor -- start 2024-03-29T11:37:44.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.849+0000 7f9c038a8640 1 -- start start 2024-03-29T11:37:44.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.849+0000 7f9c038a8640 10 monclient: init 2024-03-29T11:37:44.853 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.850+0000 7f9c038a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:44.854 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.850+0000 7f9c038a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:37:44.854 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.850+0000 7f9c038a8640 10 monclient(hunting): picked mon.noname-c con 0x7f9bfc0a9ac0 addr v1:172.21.15.89:6789/0 2024-03-29T11:37:44.854 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.850+0000 7f9c038a8640 10 monclient(hunting): picked mon.noname-b con 0x7f9bfc13ee90 addr v1:172.21.15.72:6790/0 2024-03-29T11:37:44.854 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.850+0000 7f9c038a8640 10 monclient(hunting): picked mon.noname-a con 0x7f9bfc0af900 addr v1:172.21.15.72:6789/0 2024-03-29T11:37:44.854 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.850+0000 7f9c038a8640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bfc0afd10 con 0x7f9bfc0af900 2024-03-29T11:37:44.855 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.850+0000 7f9c038a8640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bfc0af6f0 con 0x7f9bfc0a9ac0 2024-03-29T11:37:44.855 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.850+0000 7f9c038a8640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bfc0aac80 con 0x7f9bfc13ee90 2024-03-29T11:37:44.855 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.850+0000 7f9c038a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:44.855 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.850+0000 7f9c038a8640 10 monclient(hunting): authenticate will time out at 1024.138286s 2024-03-29T11:37:44.855 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.854+0000 7f9c01644640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f9bfc0a9ac0 0x7f9bfc0adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:57261/0 (socket says 172.21.15.72:57261) 2024-03-29T11:37:44.855 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.854+0000 7f9c01644640 1 -- 172.21.15.72:0/1746646110 learned_addr learned my addr 172.21.15.72:0/1746646110 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:37:44.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.858+0000 7f9bf3fff640 1 -- 172.21.15.72:0/1746646110 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 679343509 0 0) 0x7f9bfc0af6f0 con 0x7f9bfc0a9ac0 2024-03-29T11:37:44.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.858+0000 7f9bf3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:37:44.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.858+0000 7f9bf3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:44.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.858+0000 7f9bf3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:44.861 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.858+0000 7f9bf3fff640 10 monclient(hunting): my global_id is 4318 2024-03-29T11:37:44.861 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.858+0000 7f9bf3fff640 1 -- 172.21.15.72:0/1746646110 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9be8003520 con 0x7f9bfc0a9ac0 2024-03-29T11:37:44.861 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.858+0000 7f9bf3fff640 1 -- 172.21.15.72:0/1746646110 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2167797912 0 0) 0x7f9bfc0aac80 con 0x7f9bfc13ee90 2024-03-29T11:37:44.861 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.858+0000 7f9bf3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:37:44.861 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.858+0000 7f9bf3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:44.861 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.858+0000 7f9bf3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:44.861 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.858+0000 7f9bf3fff640 10 monclient(hunting): my global_id is 4322 2024-03-29T11:37:44.861 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.858+0000 7f9bf3fff640 1 -- 172.21.15.72:0/1746646110 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bfc0af6f0 con 0x7f9bfc13ee90 2024-03-29T11:37:44.861 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.858+0000 7f9bf3fff640 1 -- 172.21.15.72:0/1746646110 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 388909710 0 0) 0x7f9bfc0afd10 con 0x7f9bfc0af900 2024-03-29T11:37:44.861 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.858+0000 7f9bf3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:37:44.861 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.858+0000 7f9bf3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:44.861 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.858+0000 7f9bf3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:44.861 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.858+0000 7f9bf3fff640 10 monclient(hunting): my global_id is 4326 2024-03-29T11:37:44.861 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.858+0000 7f9bf3fff640 1 -- 172.21.15.72:0/1746646110 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bfc0aac80 con 0x7f9bfc0af900 2024-03-29T11:37:44.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.858+0000 7f9bf3fff640 1 -- 172.21.15.72:0/1746646110 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2036421510 0 0) 0x7f9be8003520 con 0x7f9bfc0a9ac0 2024-03-29T11:37:44.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.858+0000 7f9bf3fff640 1 -- 172.21.15.72:0/1746646110 >> v1:172.21.15.72:6790/0 conn(0x7f9bfc13ee90 legacy=0x7f9bfc13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:44.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.858+0000 7f9bf3fff640 1 -- 172.21.15.72:0/1746646110 >> v1:172.21.15.72:6789/0 conn(0x7f9bfc0af900 legacy=0x7f9bfc149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:44.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.858+0000 7f9bf3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:37:44.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.858+0000 7f9bf3fff640 1 monclient: found mon.noname-c 2024-03-29T11:37:44.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.858+0000 7f9bf3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:44.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.858+0000 7f9bf3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.89:6789/0 2024-03-29T11:37:44.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.858+0000 7f9bf3fff640 1 -- 172.21.15.72:0/1746646110 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bfc0aaf60 con 0x7f9bfc0a9ac0 2024-03-29T11:37:44.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.858+0000 7f9bf3fff640 10 monclient: _finish_auth 0 2024-03-29T11:37:44.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.858+0000 7f9bf3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:37:44.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.858+0000 7f9bf3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:44.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.858+0000 7f9bf3fff640 1 -- 172.21.15.72:0/1746646110 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f9bec004c90 con 0x7f9bfc0a9ac0 2024-03-29T11:37:44.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.858+0000 7f9bf3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:44.863 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.858+0000 7f9bf3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:44.863 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.858+0000 7f9bf3fff640 10 monclient: dump: 2024-03-29T11:37:44.863 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:37:44.863 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:44.863 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:44.863 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:44.863 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:44.863 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:44.863 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:37:44.863 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:37:44.863 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:37:44.863 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:44.864 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.858+0000 7f9bf3fff640 1 -- 172.21.15.72:0/1746646110 <== mon.1 v1:172.21.15.89:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9bec004df0 con 0x7f9bfc0a9ac0 2024-03-29T11:37:44.864 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.858+0000 7f9bf3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:44.864 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.858+0000 7f9bf3fff640 1 -- 172.21.15.72:0/1746646110 <== mon.1 v1:172.21.15.89:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f9bec001410 con 0x7f9bfc0a9ac0 2024-03-29T11:37:44.864 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.858+0000 7f9bf3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:44.864 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.858+0000 7f9bf3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:44.864 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.858+0000 7f9bf3fff640 10 monclient: dump: 2024-03-29T11:37:44.864 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:37:44.864 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:44.864 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:44.864 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:44.864 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:44.864 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:44.864 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:37:44.864 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:37:44.864 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:37:44.864 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:44.865 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.859+0000 7f9c038a8640 5 monclient: authenticate success, global_id 4318 2024-03-29T11:37:44.865 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.859+0000 7f9c038a8640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:44.865 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.859+0000 7f9c038a8640 4 set_mon_vals no callback set 2024-03-29T11:37:44.865 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.859+0000 7f9c038a8640 10 monclient: shutdown 2024-03-29T11:37:44.865 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.859+0000 7f9c038a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:44.865 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.859+0000 7f9c038a8640 1 -- 172.21.15.72:0/1746646110 >> v1:172.21.15.89:6789/0 conn(0x7f9bfc0a9ac0 legacy=0x7f9bfc0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:44.865 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.860+0000 7f9c038a8640 1 -- 172.21.15.72:0/1746646110 shutdown_connections 2024-03-29T11:37:44.865 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.860+0000 7f9c038a8640 1 -- 172.21.15.72:0/1746646110 >> 172.21.15.72:0/1746646110 conn(0x7f9bfc0b35e0 msgr2=0x7f9bfc0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:44.865 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.860+0000 7f9c038a8640 1 -- 172.21.15.72:0/1746646110 shutdown_connections 2024-03-29T11:37:44.869 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.860+0000 7f9c038a8640 1 -- 172.21.15.72:0/1746646110 wait complete. 2024-03-29T11:37:44.869 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.861+0000 7f9c038a8640 10 monclient: build_initial_monmap 2024-03-29T11:37:44.869 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.861+0000 7f9c038a8640 10 monclient: monmap: 2024-03-29T11:37:44.869 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:37:44.869 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:44.869 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:37:44.869 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:37:44.869 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:44.869 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:44.869 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:37:44.869 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:37:44.869 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:37:44.869 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:44.869 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.862+0000 7f9c038a8640 1 librados: starting msgr at 2024-03-29T11:37:44.869 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.862+0000 7f9c038a8640 1 librados: starting objecter 2024-03-29T11:37:44.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.862+0000 7f9c038a8640 1 Processor -- start 2024-03-29T11:37:44.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.862+0000 7f9c038a8640 1 -- start start 2024-03-29T11:37:44.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.862+0000 7f9c038a8640 1 librados: setting wanted keys 2024-03-29T11:37:44.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.862+0000 7f9c038a8640 1 librados: calling monclient init 2024-03-29T11:37:44.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.862+0000 7f9c038a8640 10 monclient: init 2024-03-29T11:37:44.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.863+0000 7f9c038a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:44.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.863+0000 7f9c038a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:37:44.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.863+0000 7f9c038a8640 10 monclient(hunting): picked mon.noname-b con 0x7f9bfc0a9ac0 addr v1:172.21.15.89:6789/0 2024-03-29T11:37:44.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.863+0000 7f9c038a8640 10 monclient(hunting): picked mon.noname-c con 0x7f9bfc0af900 addr v1:172.21.15.72:6790/0 2024-03-29T11:37:44.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.863+0000 7f9c038a8640 10 monclient(hunting): picked mon.noname-a con 0x7f9bfc13ee90 addr v1:172.21.15.72:6789/0 2024-03-29T11:37:44.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.863+0000 7f9c038a8640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bfc1c7100 con 0x7f9bfc13ee90 2024-03-29T11:37:44.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.863+0000 7f9c038a8640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bfc1c7370 con 0x7f9bfc0a9ac0 2024-03-29T11:37:44.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.863+0000 7f9c038a8640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bfc072600 con 0x7f9bfc0af900 2024-03-29T11:37:44.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.863+0000 7f9c038a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:44.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.863+0000 7f9c038a8640 10 monclient(hunting): authenticate will time out at 1024.151340s 2024-03-29T11:37:44.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.866+0000 7f9c00e43640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f9bfc0af900 0x7f9bfc1c7a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:42785/0 (socket says 172.21.15.72:42785) 2024-03-29T11:37:44.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.866+0000 7f9c00e43640 1 -- 172.21.15.72:0/1794088919 learned_addr learned my addr 172.21.15.72:0/1794088919 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:37:44.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.866+0000 7f9bf1ffb640 10 client.?.objecter ms_handle_connect 0x7f9bfc0af900 2024-03-29T11:37:44.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.866+0000 7f9bf1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:44.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.866+0000 7f9bf1ffb640 1 -- 172.21.15.72:0/1794088919 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4025644363 0 0) 0x7f9bfc072600 con 0x7f9bfc0af900 2024-03-29T11:37:44.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.866+0000 7f9bf1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:37:44.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.866+0000 7f9bf1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:44.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.866+0000 7f9bf1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:44.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.866+0000 7f9bf1ffb640 10 monclient(hunting): my global_id is 4325 2024-03-29T11:37:44.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.866+0000 7f9bf1ffb640 1 -- 172.21.15.72:0/1794088919 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9be0004eb0 con 0x7f9bfc0af900 2024-03-29T11:37:44.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.866+0000 7f9bf1ffb640 1 -- 172.21.15.72:0/1794088919 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 54827764 0 0) 0x7f9be0004eb0 con 0x7f9bfc0af900 2024-03-29T11:37:44.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.866+0000 7f9bf1ffb640 1 -- 172.21.15.72:0/1794088919 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9bfc072600 con 0x7f9bfc0af900 2024-03-29T11:37:44.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.866+0000 7f9bf1ffb640 1 -- 172.21.15.72:0/1794088919 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f9bf4001c90 con 0x7f9bfc0af900 2024-03-29T11:37:44.872 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.866+0000 7f9bf1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:44.872 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.866+0000 7f9bf1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:44.872 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.866+0000 7f9bf1ffb640 10 monclient(hunting): dump: 2024-03-29T11:37:44.872 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:37:44.872 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:44.872 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:44.872 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:44.872 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:44.872 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:44.872 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:37:44.872 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:37:44.872 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:37:44.872 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:44.872 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.866+0000 7f9bf1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:44.872 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.866+0000 7f9bf1ffb640 1 -- 172.21.15.72:0/1794088919 <== mon.2 v1:172.21.15.72:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 396529329 0 0) 0x7f9bfc072600 con 0x7f9bfc0af900 2024-03-29T11:37:44.873 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.866+0000 7f9bf1ffb640 1 -- 172.21.15.72:0/1794088919 >> v1:172.21.15.89:6789/0 conn(0x7f9bfc0a9ac0 legacy=0x7f9bfc139680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:37:44.873 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.866+0000 7f9bf1ffb640 1 -- 172.21.15.72:0/1794088919 >> v1:172.21.15.72:6789/0 conn(0x7f9bfc13ee90 legacy=0x7f9bfc1c8150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:37:44.874 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.868+0000 7f9bf1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:37:44.875 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.868+0000 7f9bf1ffb640 1 monclient: found mon.c 2024-03-29T11:37:44.875 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.868+0000 7f9bf1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:44.875 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.868+0000 7f9bf1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:37:44.875 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.868+0000 7f9bf1ffb640 1 -- 172.21.15.72:0/1794088919 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bfc1c60b0 con 0x7f9bfc0af900 2024-03-29T11:37:44.875 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.868+0000 7f9bf1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:37:44.875 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.868+0000 7f9bf1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:37:44.875 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.868+0000 7f9bf1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:44.875 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.868+0000 7f9c038a8640 5 monclient: authenticate success, global_id 4325 2024-03-29T11:37:44.875 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.868+0000 7f9c038a8640 10 monclient: _renew_subs 2024-03-29T11:37:44.875 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.868+0000 7f9c038a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:37:44.875 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.868+0000 7f9c038a8640 1 -- 172.21.15.72:0/1794088919 --> v1:172.21.15.72:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bfc1c4d10 con 0x7f9bfc0af900 2024-03-29T11:37:44.875 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.868+0000 7f9c038a8640 10 client.4325.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:44.875 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.868+0000 7f9c038a8640 10 monclient: _renew_subs 2024-03-29T11:37:44.875 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.868+0000 7f9c038a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:37:44.876 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.868+0000 7f9c038a8640 1 -- 172.21.15.72:0/1794088919 --> v1:172.21.15.72:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bfc139d90 con 0x7f9bfc0af900 2024-03-29T11:37:44.876 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.868+0000 7f9c038a8640 1 librados: init done 2024-03-29T11:37:44.876 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.869+0000 7f9bf1ffb640 1 -- 172.21.15.72:0/1794088919 <== mon.2 v1:172.21.15.72:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9bf4001df0 con 0x7f9bfc0af900 2024-03-29T11:37:44.876 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.869+0000 7f9bf1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:44.876 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.869+0000 7f9bf1ffb640 1 -- 172.21.15.72:0/1794088919 <== mon.2 v1:172.21.15.72:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f9bf4003c30 con 0x7f9bfc0af900 2024-03-29T11:37:44.876 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.869+0000 7f9bf1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:44.876 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.869+0000 7f9bf1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:44.876 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.869+0000 7f9bf1ffb640 10 monclient: dump: 2024-03-29T11:37:44.876 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:37:44.876 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:44.876 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:44.876 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:44.876 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:44.876 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:44.876 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:37:44.876 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:37:44.877 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:37:44.877 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:44.877 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.870+0000 7f9bf2ffd640 4 set_mon_vals no callback set 2024-03-29T11:37:44.877 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.872+0000 7f9bf1ffb640 1 -- 172.21.15.72:0/1794088919 <== mon.2 v1:172.21.15.72:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f9bf4003e10 con 0x7f9bfc0af900 2024-03-29T11:37:44.877 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.872+0000 7f9bf1ffb640 1 -- 172.21.15.72:0/1794088919 <== mon.2 v1:172.21.15.72:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f9bf4084eb0 con 0x7f9bfc0af900 2024-03-29T11:37:44.877 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.872+0000 7f9bf1ffb640 10 client.4325.objecter ms_dispatch 0x7f9bfc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:44.877 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.872+0000 7f9bf1ffb640 3 client.4325.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:44.877 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.872+0000 7f9bf1ffb640 3 client.4325.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:44.877 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.872+0000 7f9c038a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:44.877 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.872+0000 7f9bf1ffb640 20 client.4325.objecter dump_active .. 0 homeless 2024-03-29T11:37:44.877 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.873+0000 7f9c038a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:44.877 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.873+0000 7f9c038a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:37:44.877 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.873+0000 7f9c038a8640 1 -- 172.21.15.72:0/1794088919 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bdc005080 con 0x7f9bfc0af900 2024-03-29T11:37:44.877 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.874+0000 7f9c01644640 10 monclient: get_auth_request con 0x7f9be006dca0 auth_method 0 2024-03-29T11:37:44.877 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.874+0000 7f9bf1ffb640 10 client.4325.objecter ms_handle_connect 0x7f9be006dca0 2024-03-29T11:37:44.881 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.881+0000 7f9bf1ffb640 1 -- 172.21.15.72:0/1794088919 <== mon.2 v1:172.21.15.72:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9bf40505c0 con 0x7f9bfc0af900 2024-03-29T11:37:44.881 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.881+0000 7f9bf1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:44.882 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:44.881+0000 7f9bf1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:45.201 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:45.200+0000 7f9c038a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:45.201 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:45.200+0000 7f9c038a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:45.201 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:45.200+0000 7f9c038a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:37:45.201 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:45.200+0000 7f9c038a8640 1 -- 172.21.15.72:0/1794088919 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9bdc005330 con 0x7f9bfc0af900 2024-03-29T11:37:45.201 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:45.201+0000 7f9bf1ffb640 1 -- 172.21.15.72:0/1794088919 <== mon.2 v1:172.21.15.72:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f9bf40526c0 con 0x7f9bfc0af900 2024-03-29T11:37:45.201 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:45.201+0000 7f9bf1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:45.201 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:45.201+0000 7f9bf1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:45.202 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:37:45.202 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:45.205 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:45.204+0000 7f9c038a8640 10 librados: watch_flush enter 2024-03-29T11:37:45.205 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:45.204+0000 7f9c038a8640 10 librados: watch_flush exit 2024-03-29T11:37:45.205 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:45.204+0000 7f9c038a8640 20 client.4325.objecter shutdown clearing up homeless session... 2024-03-29T11:37:45.205 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:45.204+0000 7f9c038a8640 10 client.4325.objecter successfully canceled tick 2024-03-29T11:37:45.206 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:45.205+0000 7f9c038a8640 1 -- 172.21.15.72:0/1794088919 >> v1:172.21.15.72:6816/1578866102 conn(0x7f9be006dca0 legacy=0x7f9be0070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:45.206 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:45.205+0000 7f9c038a8640 10 monclient: shutdown 2024-03-29T11:37:45.206 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:45.205+0000 7f9c038a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:45.206 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:45.205+0000 7f9c038a8640 1 -- 172.21.15.72:0/1794088919 >> v1:172.21.15.72:6790/0 conn(0x7f9bfc0af900 legacy=0x7f9bfc1c7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:45.206 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:45.205+0000 7f9c038a8640 1 -- 172.21.15.72:0/1794088919 shutdown_connections 2024-03-29T11:37:45.206 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:45.205+0000 7f9c038a8640 1 -- 172.21.15.72:0/1794088919 >> 172.21.15.72:0/1794088919 conn(0x7f9bfc0b35e0 msgr2=0x7f9bfc13b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:45.206 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:45.205+0000 7f9c038a8640 1 -- 172.21.15.72:0/1794088919 shutdown_connections 2024-03-29T11:37:45.206 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:45.205+0000 7f9c038a8640 1 -- 172.21.15.72:0/1794088919 wait complete. 2024-03-29T11:37:45.206 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:45.205+0000 7f9c038a8640 1 librados: shutdown 2024-03-29T11:37:45.223 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:52.029 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:52.184 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.184+0000 7f06cffff640 10 monclient: get_monmap_and_config 2024-03-29T11:37:52.184 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.184+0000 7f06cffff640 10 monclient: build_initial_monmap 2024-03-29T11:37:52.184 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.184+0000 7f06cffff640 10 monclient: monmap: 2024-03-29T11:37:52.184 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:37:52.184 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:52.184 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:37:52.185213+0000 2024-03-29T11:37:52.185 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:37:52.185213+0000 2024-03-29T11:37:52.185 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:52.185 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:52.185 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:37:52.185 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:37:52.185 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:37:52.185 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:52.186 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.185+0000 7f06cffff640 1 Processor -- start 2024-03-29T11:37:52.186 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.185+0000 7f06cffff640 1 -- start start 2024-03-29T11:37:52.186 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.185+0000 7f06cffff640 10 monclient: init 2024-03-29T11:37:52.186 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.185+0000 7f06cffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:52.186 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.185+0000 7f06cffff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:37:52.186 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.185+0000 7f06cffff640 10 monclient(hunting): picked mon.noname-c con 0x7f06d00b2790 addr v1:172.21.15.89:6789/0 2024-03-29T11:37:52.186 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.185+0000 7f06cffff640 10 monclient(hunting): picked mon.noname-b con 0x7f06d0143c80 addr v1:172.21.15.72:6790/0 2024-03-29T11:37:52.186 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.186+0000 7f06cffff640 10 monclient(hunting): picked mon.noname-a con 0x7f06d0146e30 addr v1:172.21.15.72:6789/0 2024-03-29T11:37:52.186 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.186+0000 7f06cffff640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06d0146290 con 0x7f06d0146e30 2024-03-29T11:37:52.186 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.186+0000 7f06cffff640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06d00b3970 con 0x7f06d00b2790 2024-03-29T11:37:52.186 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.186+0000 7f06cffff640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06d006ee80 con 0x7f06d0143c80 2024-03-29T11:37:52.186 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.186+0000 7f06cffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:52.187 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.186+0000 7f06cffff640 10 monclient(hunting): authenticate will time out at 1031.474044s 2024-03-29T11:37:52.189 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.189+0000 7f06ce7fc640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f06d0143c80 0x7f06d0146720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:37017/0 (socket says 172.21.15.72:37017) 2024-03-29T11:37:52.189 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.189+0000 7f06ce7fc640 1 -- 172.21.15.72:0/3915990160 learned_addr learned my addr 172.21.15.72:0/3915990160 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:37:52.194 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.193+0000 7f06cdffb640 1 -- 172.21.15.72:0/3915990160 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4063625989 0 0) 0x7f06d006ee80 con 0x7f06d0143c80 2024-03-29T11:37:52.194 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.193+0000 7f06cdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:37:52.194 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.193+0000 7f06cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:52.195 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.193+0000 7f06cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:52.195 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.193+0000 7f06cdffb640 10 monclient(hunting): my global_id is 4328 2024-03-29T11:37:52.195 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.193+0000 7f06cdffb640 1 -- 172.21.15.72:0/3915990160 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06bc003520 con 0x7f06d0143c80 2024-03-29T11:37:52.195 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.194+0000 7f06cdffb640 1 -- 172.21.15.72:0/3915990160 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2913840757 0 0) 0x7f06bc003520 con 0x7f06d0143c80 2024-03-29T11:37:52.195 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.194+0000 7f06cdffb640 1 -- 172.21.15.72:0/3915990160 >> v1:172.21.15.89:6789/0 conn(0x7f06d00b2790 legacy=0x7f06d0143570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:37:52.195 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.194+0000 7f06cdffb640 1 -- 172.21.15.72:0/3915990160 >> v1:172.21.15.72:6789/0 conn(0x7f06d0146e30 legacy=0x7f06d0151bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:37:52.195 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.194+0000 7f06cdffb640 10 monclient: _finish_hunting 0 2024-03-29T11:37:52.195 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.194+0000 7f06cdffb640 1 monclient: found mon.noname-b 2024-03-29T11:37:52.195 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.194+0000 7f06cdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:52.196 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.194+0000 7f06cdffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.72:6790/0 2024-03-29T11:37:52.196 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.194+0000 7f06cdffb640 1 -- 172.21.15.72:0/3915990160 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06d006f160 con 0x7f06d0143c80 2024-03-29T11:37:52.196 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.194+0000 7f06cdffb640 10 monclient: _finish_auth 0 2024-03-29T11:37:52.196 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.194+0000 7f06cdffb640 10 monclient: _check_auth_tickets 2024-03-29T11:37:52.196 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.194+0000 7f06cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:52.196 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.194+0000 7f06cffff640 5 monclient: authenticate success, global_id 4328 2024-03-29T11:37:52.196 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.194+0000 7f06cffff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:52.196 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.194+0000 7f06cdffb640 1 -- 172.21.15.72:0/3915990160 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f06b8001670 con 0x7f06d0143c80 2024-03-29T11:37:52.196 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.194+0000 7f06cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:52.196 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.194+0000 7f06cdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:52.197 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.194+0000 7f06cdffb640 10 monclient: dump: 2024-03-29T11:37:52.197 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:37:52.197 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:52.197 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:52.197 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:52.197 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:52.197 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:52.197 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:37:52.197 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:37:52.197 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:37:52.197 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:52.197 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.194+0000 7f06cffff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:52.197 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.195+0000 7f06cdffb640 1 -- 172.21.15.72:0/3915990160 <== mon.2 v1:172.21.15.72:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06b80018d0 con 0x7f06d0143c80 2024-03-29T11:37:52.197 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.195+0000 7f06cdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:52.197 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.195+0000 7f06cffff640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:52.197 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.195+0000 7f06cffff640 4 set_mon_vals no callback set 2024-03-29T11:37:52.198 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.196+0000 7f06cffff640 10 monclient: shutdown 2024-03-29T11:37:52.198 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.196+0000 7f06cffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:52.198 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.196+0000 7f06cffff640 1 -- 172.21.15.72:0/3915990160 >> v1:172.21.15.72:6790/0 conn(0x7f06d0143c80 legacy=0x7f06d0146720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:52.198 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.196+0000 7f06cffff640 1 -- 172.21.15.72:0/3915990160 shutdown_connections 2024-03-29T11:37:52.198 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.196+0000 7f06cffff640 1 -- 172.21.15.72:0/3915990160 >> 172.21.15.72:0/3915990160 conn(0x7f06d006d070 msgr2=0x7f06d0066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:52.199 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.199+0000 7f06cffff640 1 -- 172.21.15.72:0/3915990160 shutdown_connections 2024-03-29T11:37:52.202 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.199+0000 7f06cffff640 1 -- 172.21.15.72:0/3915990160 wait complete. 2024-03-29T11:37:52.202 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.200+0000 7f06cffff640 10 monclient: build_initial_monmap 2024-03-29T11:37:52.202 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.200+0000 7f06cffff640 10 monclient: monmap: 2024-03-29T11:37:52.202 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:37:52.202 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:52.202 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:37:52.202 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:37:52.203 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:52.203 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:52.203 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:37:52.203 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:37:52.203 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:37:52.203 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:52.203 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.201+0000 7f06cffff640 1 librados: starting msgr at 2024-03-29T11:37:52.203 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.201+0000 7f06cffff640 1 librados: starting objecter 2024-03-29T11:37:52.203 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.201+0000 7f06cffff640 1 Processor -- start 2024-03-29T11:37:52.207 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.201+0000 7f06cffff640 1 -- start start 2024-03-29T11:37:52.207 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.201+0000 7f06cffff640 1 librados: setting wanted keys 2024-03-29T11:37:52.207 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.201+0000 7f06cffff640 1 librados: calling monclient init 2024-03-29T11:37:52.207 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.201+0000 7f06cffff640 10 monclient: init 2024-03-29T11:37:52.207 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.202+0000 7f06cffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:52.207 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.202+0000 7f06cffff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:37:52.207 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.202+0000 7f06cffff640 10 monclient(hunting): picked mon.noname-b con 0x7f06d00b2790 addr v1:172.21.15.89:6789/0 2024-03-29T11:37:52.207 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.202+0000 7f06cffff640 10 monclient(hunting): picked mon.noname-c con 0x7f06d0143c80 addr v1:172.21.15.72:6790/0 2024-03-29T11:37:52.207 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.202+0000 7f06cffff640 10 monclient(hunting): picked mon.noname-a con 0x7f06d0146e30 addr v1:172.21.15.72:6789/0 2024-03-29T11:37:52.207 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.202+0000 7f06cffff640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06d0147ba0 con 0x7f06d0146e30 2024-03-29T11:37:52.207 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.202+0000 7f06cffff640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06d0147610 con 0x7f06d00b2790 2024-03-29T11:37:52.207 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.202+0000 7f06cffff640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06d01477a0 con 0x7f06d0143c80 2024-03-29T11:37:52.207 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.202+0000 7f06cffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:52.207 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.202+0000 7f06cffff640 10 monclient(hunting): authenticate will time out at 1031.490605s 2024-03-29T11:37:52.208 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.205+0000 7f06cf7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f06d0146e30 0x7f06d01e5aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45575/0 (socket says 172.21.15.72:45575) 2024-03-29T11:37:52.208 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.205+0000 7f06cf7fe640 1 -- 172.21.15.72:0/531977184 learned_addr learned my addr 172.21.15.72:0/531977184 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:37:52.208 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.205+0000 7f06c77fe640 10 client.?.objecter ms_handle_connect 0x7f06d0146e30 2024-03-29T11:37:52.208 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.205+0000 7f06c77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:52.208 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.205+0000 7f06c77fe640 1 -- 172.21.15.72:0/531977184 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 966733246 0 0) 0x7f06d0147ba0 con 0x7f06d0146e30 2024-03-29T11:37:52.208 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.205+0000 7f06c77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:52.208 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.205+0000 7f06c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:52.208 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.205+0000 7f06c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:52.208 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.205+0000 7f06c77fe640 10 monclient(hunting): my global_id is 4329 2024-03-29T11:37:52.208 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.205+0000 7f06c77fe640 1 -- 172.21.15.72:0/531977184 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06b0004eb0 con 0x7f06d0146e30 2024-03-29T11:37:52.209 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.206+0000 7f06c77fe640 1 -- 172.21.15.72:0/531977184 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1466713849 0 0) 0x7f06b0004eb0 con 0x7f06d0146e30 2024-03-29T11:37:52.209 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.206+0000 7f06c77fe640 1 -- 172.21.15.72:0/531977184 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06d0147ba0 con 0x7f06d0146e30 2024-03-29T11:37:52.209 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.206+0000 7f06c77fe640 1 -- 172.21.15.72:0/531977184 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f06c8001ad0 con 0x7f06d0146e30 2024-03-29T11:37:52.209 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.206+0000 7f06c77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:52.209 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.206+0000 7f06c77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:52.209 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.206+0000 7f06c77fe640 10 monclient(hunting): dump: 2024-03-29T11:37:52.209 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:37:52.209 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:52.209 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:52.209 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:52.209 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:52.210 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:52.210 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:37:52.210 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:37:52.210 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:37:52.210 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:52.210 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.206+0000 7f06c77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:52.210 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.206+0000 7f06c77fe640 1 -- 172.21.15.72:0/531977184 <== mon.0 v1:172.21.15.72:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2993601403 0 0) 0x7f06d0147ba0 con 0x7f06d0146e30 2024-03-29T11:37:52.210 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.206+0000 7f06c77fe640 1 -- 172.21.15.72:0/531977184 >> v1:172.21.15.72:6790/0 conn(0x7f06d0143c80 legacy=0x7f06d01deef0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:37:52.211 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.210+0000 7f06c77fe640 1 -- 172.21.15.72:0/531977184 >> v1:172.21.15.89:6789/0 conn(0x7f06d00b2790 legacy=0x7f06d01de7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:52.211 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.210+0000 7f06c77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:37:52.212 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.210+0000 7f06c77fe640 1 monclient: found mon.a 2024-03-29T11:37:52.212 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.210+0000 7f06c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:52.212 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.210+0000 7f06c77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:37:52.212 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.210+0000 7f06c77fe640 1 -- 172.21.15.72:0/531977184 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06d0147930 con 0x7f06d0146e30 2024-03-29T11:37:52.212 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.210+0000 7f06c77fe640 10 monclient: _finish_auth 0 2024-03-29T11:37:52.212 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.210+0000 7f06c77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:37:52.212 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.210+0000 7f06c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:52.215 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.210+0000 7f06cffff640 5 monclient: authenticate success, global_id 4329 2024-03-29T11:37:52.215 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.210+0000 7f06cffff640 10 monclient: _renew_subs 2024-03-29T11:37:52.215 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.210+0000 7f06cffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:37:52.215 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.210+0000 7f06cffff640 1 -- 172.21.15.72:0/531977184 --> v1:172.21.15.72:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06d00b69d0 con 0x7f06d0146e30 2024-03-29T11:37:52.215 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.210+0000 7f06cffff640 10 client.4329.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:52.215 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.210+0000 7f06cffff640 10 monclient: _renew_subs 2024-03-29T11:37:52.215 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.210+0000 7f06cffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:37:52.215 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.210+0000 7f06cffff640 1 -- 172.21.15.72:0/531977184 --> v1:172.21.15.72:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f06d01454c0 con 0x7f06d0146e30 2024-03-29T11:37:52.215 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.210+0000 7f06cffff640 1 librados: init done 2024-03-29T11:37:52.216 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.211+0000 7f06c77fe640 1 -- 172.21.15.72:0/531977184 <== mon.0 v1:172.21.15.72:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06c8001c30 con 0x7f06d0146e30 2024-03-29T11:37:52.216 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.211+0000 7f06c77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:52.216 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.211+0000 7f06c77fe640 1 -- 172.21.15.72:0/531977184 <== mon.0 v1:172.21.15.72:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f06c8004d80 con 0x7f06d0146e30 2024-03-29T11:37:52.216 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.211+0000 7f06c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:52.216 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.211+0000 7f06c77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:52.216 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.211+0000 7f06c77fe640 10 monclient: dump: 2024-03-29T11:37:52.216 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:37:52.216 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:52.216 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:52.216 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:52.216 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:52.216 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:52.216 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:37:52.216 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:37:52.216 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:37:52.217 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:52.217 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.211+0000 7f06c57fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:52.217 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.211+0000 7f06c57fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:52.217 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.211+0000 7f06c57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:37:52.217 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.211+0000 7f06ccff9640 4 set_mon_vals no callback set 2024-03-29T11:37:52.217 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.211+0000 7f06c57fa640 1 -- 172.21.15.72:0/531977184 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06d004f730 con 0x7f06d0146e30 2024-03-29T11:37:52.217 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.214+0000 7f06c77fe640 1 -- 172.21.15.72:0/531977184 <== mon.0 v1:172.21.15.72:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f06c8004ee0 con 0x7f06d0146e30 2024-03-29T11:37:52.217 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.214+0000 7f06c77fe640 1 -- 172.21.15.72:0/531977184 <== mon.0 v1:172.21.15.72:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f06c8084d60 con 0x7f06d0146e30 2024-03-29T11:37:52.217 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.214+0000 7f06c77fe640 10 client.4329.objecter ms_dispatch 0x7f06d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:52.217 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.214+0000 7f06c77fe640 3 client.4329.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:52.217 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.214+0000 7f06c77fe640 3 client.4329.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:52.217 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.214+0000 7f06c77fe640 20 client.4329.objecter dump_active .. 0 homeless 2024-03-29T11:37:52.218 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.216+0000 7f06ceffd640 10 monclient: get_auth_request con 0x7f06b006dc90 auth_method 0 2024-03-29T11:37:52.218 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.216+0000 7f06c77fe640 10 client.4329.objecter ms_handle_connect 0x7f06b006dc90 2024-03-29T11:37:52.221 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.221+0000 7f06c77fe640 1 -- 172.21.15.72:0/531977184 <== mon.0 v1:172.21.15.72:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f06c8050640 con 0x7f06d0146e30 2024-03-29T11:37:52.222 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.221+0000 7f06c77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:52.222 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.221+0000 7f06c77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:52.545 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.545+0000 7f06c57fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:52.545 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.545+0000 7f06c57fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:52.545 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.545+0000 7f06c57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:37:52.545 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.545+0000 7f06c57fa640 1 -- 172.21.15.72:0/531977184 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06d01457c0 con 0x7f06d0146e30 2024-03-29T11:37:52.546 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.546+0000 7f06c77fe640 1 -- 172.21.15.72:0/531977184 <== mon.0 v1:172.21.15.72:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f06c8052570 con 0x7f06d0146e30 2024-03-29T11:37:52.546 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.546+0000 7f06c77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:52.546 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.546+0000 7f06c77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:52.549 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:37:52.549 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:52.552 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.552+0000 7f06c57fa640 10 librados: watch_flush enter 2024-03-29T11:37:52.552 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.552+0000 7f06c57fa640 10 librados: watch_flush exit 2024-03-29T11:37:52.552 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.552+0000 7f06c57fa640 20 client.4329.objecter shutdown clearing up homeless session... 2024-03-29T11:37:52.552 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.552+0000 7f06c57fa640 10 client.4329.objecter successfully canceled tick 2024-03-29T11:37:52.552 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.552+0000 7f06c57fa640 1 -- 172.21.15.72:0/531977184 >> v1:172.21.15.72:6816/1578866102 conn(0x7f06b006dc90 legacy=0x7f06b0070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:52.552 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.552+0000 7f06c57fa640 10 monclient: shutdown 2024-03-29T11:37:52.552 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.552+0000 7f06c57fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:52.553 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.552+0000 7f06c57fa640 1 -- 172.21.15.72:0/531977184 >> v1:172.21.15.72:6789/0 conn(0x7f06d0146e30 legacy=0x7f06d01e5aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:52.553 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.552+0000 7f06c57fa640 1 -- 172.21.15.72:0/531977184 shutdown_connections 2024-03-29T11:37:52.553 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.552+0000 7f06c57fa640 1 -- 172.21.15.72:0/531977184 >> 172.21.15.72:0/531977184 conn(0x7f06d006d070 msgr2=0x7f06d0069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:52.555 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.554+0000 7f06c57fa640 1 -- 172.21.15.72:0/531977184 shutdown_connections 2024-03-29T11:37:52.555 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.555+0000 7f06c57fa640 1 -- 172.21.15.72:0/531977184 wait complete. 2024-03-29T11:37:52.556 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:52.556+0000 7f06c57fa640 1 librados: shutdown 2024-03-29T11:37:52.576 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:59.383 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:59.533 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.532+0000 7f8bcd352640 10 monclient: get_monmap_and_config 2024-03-29T11:37:59.533 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.532+0000 7f8bcd352640 10 monclient: build_initial_monmap 2024-03-29T11:37:59.533 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.532+0000 7f8bcd352640 10 monclient: monmap: 2024-03-29T11:37:59.533 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:37:59.533 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:59.534 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:37:59.533463+0000 2024-03-29T11:37:59.534 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:37:59.533463+0000 2024-03-29T11:37:59.534 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:59.534 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:59.534 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:37:59.534 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:37:59.534 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:37:59.534 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:59.534 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.533+0000 7f8bcd352640 1 Processor -- start 2024-03-29T11:37:59.534 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.533+0000 7f8bcd352640 1 -- start start 2024-03-29T11:37:59.534 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.533+0000 7f8bcd352640 10 monclient: init 2024-03-29T11:37:59.535 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.534+0000 7f8bcd352640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:59.535 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.534+0000 7f8bcd352640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:37:59.537 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.535+0000 7f8bcd352640 10 monclient(hunting): picked mon.noname-c con 0x7f8bc80a9ac0 addr v1:172.21.15.89:6789/0 2024-03-29T11:37:59.538 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.535+0000 7f8bcd352640 10 monclient(hunting): picked mon.noname-b con 0x7f8bc813ee90 addr v1:172.21.15.72:6790/0 2024-03-29T11:37:59.538 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.535+0000 7f8bcd352640 10 monclient(hunting): picked mon.noname-a con 0x7f8bc80af900 addr v1:172.21.15.72:6789/0 2024-03-29T11:37:59.538 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.535+0000 7f8bcd352640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bc80afd10 con 0x7f8bc80af900 2024-03-29T11:37:59.538 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.535+0000 7f8bcd352640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bc80af6f0 con 0x7f8bc80a9ac0 2024-03-29T11:37:59.538 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.535+0000 7f8bcd352640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bc80aac80 con 0x7f8bc813ee90 2024-03-29T11:37:59.538 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.535+0000 7f8bcd352640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:59.538 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.535+0000 7f8bcd352640 10 monclient(hunting): authenticate will time out at 1038.823253s 2024-03-29T11:37:59.540 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.539+0000 7f8bc6ffd640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f8bc80a9ac0 0x7f8bc80adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:38403/0 (socket says 172.21.15.72:38403) 2024-03-29T11:37:59.540 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.540+0000 7f8bc6ffd640 1 -- 172.21.15.72:0/383039427 learned_addr learned my addr 172.21.15.72:0/383039427 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:37:59.540 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.540+0000 7f8bc5ffb640 1 -- 172.21.15.72:0/383039427 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1778172071 0 0) 0x7f8bc80aac80 con 0x7f8bc813ee90 2024-03-29T11:37:59.541 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.540+0000 7f8bc5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:37:59.541 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.540+0000 7f8bc5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:59.541 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.540+0000 7f8bc5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:59.541 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.540+0000 7f8bc5ffb640 10 monclient(hunting): my global_id is 4331 2024-03-29T11:37:59.541 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.540+0000 7f8bc5ffb640 1 -- 172.21.15.72:0/383039427 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bb0003520 con 0x7f8bc813ee90 2024-03-29T11:37:59.544 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.541+0000 7f8bc5ffb640 1 -- 172.21.15.72:0/383039427 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3944162278 0 0) 0x7f8bb0003520 con 0x7f8bc813ee90 2024-03-29T11:37:59.544 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.541+0000 7f8bc5ffb640 1 -- 172.21.15.72:0/383039427 >> v1:172.21.15.89:6789/0 conn(0x7f8bc80a9ac0 legacy=0x7f8bc80adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:59.544 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.541+0000 7f8bc5ffb640 1 -- 172.21.15.72:0/383039427 >> v1:172.21.15.72:6789/0 conn(0x7f8bc80af900 legacy=0x7f8bc8149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:37:59.544 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.541+0000 7f8bc5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:37:59.544 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.541+0000 7f8bc5ffb640 1 monclient: found mon.noname-b 2024-03-29T11:37:59.544 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.541+0000 7f8bc5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:59.544 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.541+0000 7f8bc5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.72:6790/0 2024-03-29T11:37:59.544 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.541+0000 7f8bc5ffb640 1 -- 172.21.15.72:0/383039427 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc80aaf60 con 0x7f8bc813ee90 2024-03-29T11:37:59.544 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.542+0000 7f8bc5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:37:59.544 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.542+0000 7f8bc5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:37:59.544 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.542+0000 7f8bc5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:59.544 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.542+0000 7f8bc5ffb640 1 -- 172.21.15.72:0/383039427 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f8bbc001690 con 0x7f8bc813ee90 2024-03-29T11:37:59.544 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.542+0000 7f8bc5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:59.544 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.542+0000 7f8bc5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:59.544 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.542+0000 7f8bc5ffb640 10 monclient: dump: 2024-03-29T11:37:59.545 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:37:59.545 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:59.545 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:59.545 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:59.545 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:59.545 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:59.545 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:37:59.545 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:37:59.545 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:37:59.545 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:59.545 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.542+0000 7f8bc5ffb640 1 -- 172.21.15.72:0/383039427 <== mon.2 v1:172.21.15.72:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8bbc001920 con 0x7f8bc813ee90 2024-03-29T11:37:59.545 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.542+0000 7f8bc5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:59.545 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.542+0000 7f8bc5ffb640 1 -- 172.21.15.72:0/383039427 <== mon.2 v1:172.21.15.72:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f8bbc001cb0 con 0x7f8bc813ee90 2024-03-29T11:37:59.545 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.542+0000 7f8bc5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:59.545 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.542+0000 7f8bc5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:59.546 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.542+0000 7f8bc5ffb640 10 monclient: dump: 2024-03-29T11:37:59.546 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:37:59.546 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:59.546 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:59.546 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:59.546 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:59.546 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:59.546 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:37:59.546 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:37:59.546 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:37:59.546 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:59.546 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.543+0000 7f8bcd352640 5 monclient: authenticate success, global_id 4331 2024-03-29T11:37:59.546 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.543+0000 7f8bcd352640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:59.547 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.543+0000 7f8bcd352640 4 set_mon_vals no callback set 2024-03-29T11:37:59.547 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.544+0000 7f8bcd352640 10 monclient: shutdown 2024-03-29T11:37:59.548 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.544+0000 7f8bcd352640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:59.548 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.544+0000 7f8bcd352640 1 -- 172.21.15.72:0/383039427 >> v1:172.21.15.72:6790/0 conn(0x7f8bc813ee90 legacy=0x7f8bc813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:59.548 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.545+0000 7f8bcd352640 1 -- 172.21.15.72:0/383039427 shutdown_connections 2024-03-29T11:37:59.548 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.545+0000 7f8bcd352640 1 -- 172.21.15.72:0/383039427 >> 172.21.15.72:0/383039427 conn(0x7f8bc80b35e0 msgr2=0x7f8bc80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:59.548 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.546+0000 7f8bcd352640 1 -- 172.21.15.72:0/383039427 shutdown_connections 2024-03-29T11:37:59.548 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.546+0000 7f8bcd352640 1 -- 172.21.15.72:0/383039427 wait complete. 2024-03-29T11:37:59.548 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.547+0000 7f8bcd352640 10 monclient: build_initial_monmap 2024-03-29T11:37:59.548 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.547+0000 7f8bcd352640 10 monclient: monmap: 2024-03-29T11:37:59.548 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:37:59.548 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:59.548 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:37:59.548 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:37:59.548 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:59.548 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:59.549 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:37:59.549 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:37:59.549 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:37:59.549 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:59.550 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.548+0000 7f8bcd352640 1 librados: starting msgr at 2024-03-29T11:37:59.550 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.548+0000 7f8bcd352640 1 librados: starting objecter 2024-03-29T11:37:59.550 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.548+0000 7f8bcd352640 1 Processor -- start 2024-03-29T11:37:59.550 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.549+0000 7f8bcd352640 1 -- start start 2024-03-29T11:37:59.550 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.549+0000 7f8bcd352640 1 librados: setting wanted keys 2024-03-29T11:37:59.550 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.549+0000 7f8bcd352640 1 librados: calling monclient init 2024-03-29T11:37:59.550 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.549+0000 7f8bcd352640 10 monclient: init 2024-03-29T11:37:59.550 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.550+0000 7f8bcd352640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:59.550 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.550+0000 7f8bcd352640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:37:59.552 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.552+0000 7f8bcd352640 10 monclient(hunting): picked mon.noname-a con 0x7f8bc80a9ac0 addr v1:172.21.15.72:6789/0 2024-03-29T11:37:59.553 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.552+0000 7f8bcd352640 10 monclient(hunting): picked mon.noname-b con 0x7f8bc80af900 addr v1:172.21.15.89:6789/0 2024-03-29T11:37:59.554 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.554+0000 7f8bcd352640 10 monclient(hunting): picked mon.noname-c con 0x7f8bc813ee90 addr v1:172.21.15.72:6790/0 2024-03-29T11:37:59.554 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.554+0000 7f8bcd352640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bc813a1c0 con 0x7f8bc80a9ac0 2024-03-29T11:37:59.554 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.554+0000 7f8bcd352640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bc806fc00 con 0x7f8bc80af900 2024-03-29T11:37:59.554 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.554+0000 7f8bcd352640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bc806fee0 con 0x7f8bc813ee90 2024-03-29T11:37:59.554 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.554+0000 7f8bcd352640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:59.554 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.554+0000 7f8bcd352640 10 monclient(hunting): authenticate will time out at 1038.842190s 2024-03-29T11:37:59.555 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.554+0000 7f8bc6ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f8bc80a9ac0 0x7f8bc81d28f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:57275/0 (socket says 172.21.15.72:57275) 2024-03-29T11:37:59.555 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.554+0000 7f8bc6ffd640 1 -- 172.21.15.72:0/3927461392 learned_addr learned my addr 172.21.15.72:0/3927461392 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:37:59.556 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.555+0000 7f8baf7fe640 10 client.?.objecter ms_handle_connect 0x7f8bc80a9ac0 2024-03-29T11:37:59.556 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.556+0000 7f8baf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:59.556 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.556+0000 7f8baf7fe640 1 -- 172.21.15.72:0/3927461392 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 16373049 0 0) 0x7f8bc813a1c0 con 0x7f8bc80a9ac0 2024-03-29T11:37:59.556 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.556+0000 7f8baf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:59.556 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.556+0000 7f8baf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:59.556 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.556+0000 7f8baf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:59.556 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.556+0000 7f8baf7fe640 10 monclient(hunting): my global_id is 4332 2024-03-29T11:37:59.556 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.556+0000 7f8baf7fe640 1 -- 172.21.15.72:0/3927461392 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bb4004eb0 con 0x7f8bc80a9ac0 2024-03-29T11:37:59.564 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.558+0000 7f8baf7fe640 10 client.?.objecter ms_handle_connect 0x7f8bc80af900 2024-03-29T11:37:59.564 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.558+0000 7f8baf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:59.564 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.558+0000 7f8baf7fe640 10 client.?.objecter ms_handle_connect 0x7f8bc813ee90 2024-03-29T11:37:59.564 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.558+0000 7f8baf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:59.564 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.558+0000 7f8baf7fe640 1 -- 172.21.15.72:0/3927461392 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1100576177 0 0) 0x7f8bc806fc00 con 0x7f8bc80af900 2024-03-29T11:37:59.564 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.558+0000 7f8baf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:59.564 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.558+0000 7f8baf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:59.565 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.558+0000 7f8baf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:59.565 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.558+0000 7f8baf7fe640 10 monclient(hunting): my global_id is 4324 2024-03-29T11:37:59.565 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.559+0000 7f8baf7fe640 1 -- 172.21.15.72:0/3927461392 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bc813a1c0 con 0x7f8bc80af900 2024-03-29T11:37:59.565 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.559+0000 7f8baf7fe640 1 -- 172.21.15.72:0/3927461392 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 695589797 0 0) 0x7f8bc806fee0 con 0x7f8bc813ee90 2024-03-29T11:37:59.565 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.559+0000 7f8baf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:59.565 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.559+0000 7f8baf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:59.565 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.559+0000 7f8baf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:59.565 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.559+0000 7f8baf7fe640 10 monclient(hunting): my global_id is 4334 2024-03-29T11:37:59.565 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.559+0000 7f8baf7fe640 1 -- 172.21.15.72:0/3927461392 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bc806fc00 con 0x7f8bc813ee90 2024-03-29T11:37:59.565 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.559+0000 7f8baf7fe640 1 -- 172.21.15.72:0/3927461392 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 54786031 0 0) 0x7f8bb4004eb0 con 0x7f8bc80a9ac0 2024-03-29T11:37:59.565 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.559+0000 7f8baf7fe640 1 -- 172.21.15.72:0/3927461392 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8bc806fee0 con 0x7f8bc80a9ac0 2024-03-29T11:37:59.565 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.559+0000 7f8baf7fe640 1 -- 172.21.15.72:0/3927461392 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f8bb8001950 con 0x7f8bc80a9ac0 2024-03-29T11:37:59.565 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.559+0000 7f8baf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:59.565 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.559+0000 7f8baf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:59.565 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.559+0000 7f8baf7fe640 10 monclient(hunting): dump: 2024-03-29T11:37:59.566 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:37:59.566 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:59.566 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:59.566 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:59.566 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:59.566 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:59.566 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:37:59.566 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:37:59.566 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:37:59.566 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:59.566 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.559+0000 7f8baf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:59.566 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.559+0000 7f8baf7fe640 1 -- 172.21.15.72:0/3927461392 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 849995296 0 0) 0x7f8bc806fc00 con 0x7f8bc813ee90 2024-03-29T11:37:59.566 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.559+0000 7f8baf7fe640 1 -- 172.21.15.72:0/3927461392 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8bb4004eb0 con 0x7f8bc813ee90 2024-03-29T11:37:59.566 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.559+0000 7f8baf7fe640 1 -- 172.21.15.72:0/3927461392 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f8bc00019e0 con 0x7f8bc813ee90 2024-03-29T11:37:59.567 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.559+0000 7f8baf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:59.567 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.559+0000 7f8baf7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:59.567 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.559+0000 7f8baf7fe640 10 monclient(hunting): dump: 2024-03-29T11:37:59.567 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:37:59.567 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:59.567 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:59.567 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:59.567 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:59.567 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:59.567 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:37:59.567 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:37:59.567 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:37:59.567 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:59.567 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.559+0000 7f8baf7fe640 1 -- 172.21.15.72:0/3927461392 <== mon.0 v1:172.21.15.72:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3636355008 0 0) 0x7f8bc806fee0 con 0x7f8bc80a9ac0 2024-03-29T11:37:59.567 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.559+0000 7f8baf7fe640 1 -- 172.21.15.72:0/3927461392 >> v1:172.21.15.72:6790/0 conn(0x7f8bc813ee90 legacy=0x7f8bc8139ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:59.567 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.559+0000 7f8baf7fe640 1 -- 172.21.15.72:0/3927461392 >> v1:172.21.15.89:6789/0 conn(0x7f8bc80af900 legacy=0x7f8bc81393a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:59.568 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.559+0000 7f8baf7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:37:59.568 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.559+0000 7f8baf7fe640 1 monclient: found mon.a 2024-03-29T11:37:59.568 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.559+0000 7f8baf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:59.568 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.559+0000 7f8baf7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:37:59.568 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.559+0000 7f8baf7fe640 1 -- 172.21.15.72:0/3927461392 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc80701c0 con 0x7f8bc80a9ac0 2024-03-29T11:37:59.568 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.559+0000 7f8baf7fe640 10 monclient: _finish_auth 0 2024-03-29T11:37:59.568 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.559+0000 7f8baf7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:37:59.568 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.559+0000 7f8baf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:59.568 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.559+0000 7f8bcd352640 5 monclient: authenticate success, global_id 4332 2024-03-29T11:37:59.568 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.559+0000 7f8bcd352640 10 monclient: _renew_subs 2024-03-29T11:37:59.568 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.559+0000 7f8bcd352640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:37:59.568 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.559+0000 7f8bcd352640 1 -- 172.21.15.72:0/3927461392 --> v1:172.21.15.72:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bc806f390 con 0x7f8bc80a9ac0 2024-03-29T11:37:59.568 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.560+0000 7f8bcd352640 10 client.4332.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:59.568 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.560+0000 7f8bcd352640 10 monclient: _renew_subs 2024-03-29T11:37:59.568 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.560+0000 7f8bcd352640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:37:59.569 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.560+0000 7f8bcd352640 1 -- 172.21.15.72:0/3927461392 --> v1:172.21.15.72:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bc806ef00 con 0x7f8bc80a9ac0 2024-03-29T11:37:59.569 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.560+0000 7f8baf7fe640 1 -- 172.21.15.72:0/3927461392 <== mon.0 v1:172.21.15.72:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8bb8001ab0 con 0x7f8bc80a9ac0 2024-03-29T11:37:59.569 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.560+0000 7f8bcd352640 1 librados: init done 2024-03-29T11:37:59.569 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.560+0000 7f8baf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:59.569 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.560+0000 7f8bc57fa640 4 set_mon_vals no callback set 2024-03-29T11:37:59.569 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.561+0000 7f8baf7fe640 1 -- 172.21.15.72:0/3927461392 <== mon.0 v1:172.21.15.72:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f8bb8004a90 con 0x7f8bc80a9ac0 2024-03-29T11:37:59.569 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.561+0000 7f8baf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:59.569 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.561+0000 7f8baf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:59.569 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.561+0000 7f8baf7fe640 10 monclient: dump: 2024-03-29T11:37:59.569 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:37:59.569 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:37:59.569 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:59.569 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:37:59.569 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:59.569 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:37:59.570 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:37:59.570 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:37:59.570 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:37:59.570 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:37:59.570 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.562+0000 7f8bcd352640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:59.570 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.562+0000 7f8bcd352640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:59.570 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.562+0000 7f8bcd352640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:37:59.570 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.562+0000 7f8bcd352640 1 -- 172.21.15.72:0/3927461392 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bc806ec80 con 0x7f8bc80a9ac0 2024-03-29T11:37:59.570 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.562+0000 7f8baf7fe640 1 -- 172.21.15.72:0/3927461392 <== mon.0 v1:172.21.15.72:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f8bb801aa60 con 0x7f8bc80a9ac0 2024-03-29T11:37:59.570 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.563+0000 7f8baf7fe640 1 -- 172.21.15.72:0/3927461392 <== mon.0 v1:172.21.15.72:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f8bb8084a90 con 0x7f8bc80a9ac0 2024-03-29T11:37:59.570 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.563+0000 7f8baf7fe640 10 client.4332.objecter ms_dispatch 0x7f8bc8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:59.570 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.563+0000 7f8baf7fe640 3 client.4332.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:59.570 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.563+0000 7f8baf7fe640 3 client.4332.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:59.570 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.564+0000 7f8baf7fe640 20 client.4332.objecter dump_active .. 0 homeless 2024-03-29T11:37:59.570 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.564+0000 7f8bc67fc640 10 monclient: get_auth_request con 0x7f8bb406e0a0 auth_method 0 2024-03-29T11:37:59.571 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.564+0000 7f8baf7fe640 10 client.4332.objecter ms_handle_connect 0x7f8bb406e0a0 2024-03-29T11:37:59.571 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.571+0000 7f8baf7fe640 1 -- 172.21.15.72:0/3927461392 <== mon.0 v1:172.21.15.72:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8bb80501a0 con 0x7f8bc80a9ac0 2024-03-29T11:37:59.571 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.571+0000 7f8baf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:59.571 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.571+0000 7f8baf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:59.889 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.889+0000 7f8bcd352640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:59.889 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.889+0000 7f8bcd352640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:59.890 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.889+0000 7f8bcd352640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:37:59.890 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.889+0000 7f8bcd352640 1 -- 172.21.15.72:0/3927461392 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8bc8075f10 con 0x7f8bc80a9ac0 2024-03-29T11:37:59.893 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.892+0000 7f8baf7fe640 1 -- 172.21.15.72:0/3927461392 <== mon.0 v1:172.21.15.72:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f8bb80522a0 con 0x7f8bc80a9ac0 2024-03-29T11:37:59.893 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.893+0000 7f8baf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:59.893 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.893+0000 7f8baf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:59.893 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:37:59.893 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:59.898 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.896+0000 7f8bcd352640 10 librados: watch_flush enter 2024-03-29T11:37:59.898 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.898+0000 7f8bcd352640 10 librados: watch_flush exit 2024-03-29T11:37:59.898 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.898+0000 7f8bcd352640 20 client.4332.objecter shutdown clearing up homeless session... 2024-03-29T11:37:59.898 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.898+0000 7f8bcd352640 10 client.4332.objecter successfully canceled tick 2024-03-29T11:37:59.899 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.899+0000 7f8bcd352640 1 -- 172.21.15.72:0/3927461392 >> v1:172.21.15.72:6816/1578866102 conn(0x7f8bb406e0a0 legacy=0x7f8bb4070560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:59.899 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.899+0000 7f8bcd352640 10 monclient: shutdown 2024-03-29T11:37:59.900 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.899+0000 7f8bcd352640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:59.900 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.899+0000 7f8bcd352640 1 -- 172.21.15.72:0/3927461392 >> v1:172.21.15.72:6789/0 conn(0x7f8bc80a9ac0 legacy=0x7f8bc81d28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:59.903 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.901+0000 7f8bcd352640 1 -- 172.21.15.72:0/3927461392 shutdown_connections 2024-03-29T11:37:59.903 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.901+0000 7f8bcd352640 1 -- 172.21.15.72:0/3927461392 >> 172.21.15.72:0/3927461392 conn(0x7f8bc80b35e0 msgr2=0x7f8bc813b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:59.903 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.903+0000 7f8bcd352640 1 -- 172.21.15.72:0/3927461392 shutdown_connections 2024-03-29T11:37:59.906 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.906+0000 7f8bcd352640 1 -- 172.21.15.72:0/3927461392 wait complete. 2024-03-29T11:37:59.907 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:37:59.906+0000 7f8bcd352640 1 librados: shutdown 2024-03-29T11:37:59.924 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:06.731 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:06.880 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.879+0000 7fc21bc6d640 10 monclient: get_monmap_and_config 2024-03-29T11:38:06.881 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.879+0000 7fc21bc6d640 10 monclient: build_initial_monmap 2024-03-29T11:38:06.881 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.879+0000 7fc21bc6d640 10 monclient: monmap: 2024-03-29T11:38:06.881 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:38:06.881 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:38:06.881 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:38:06.880511+0000 2024-03-29T11:38:06.881 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:38:06.880511+0000 2024-03-29T11:38:06.881 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:06.881 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:06.881 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:38:06.881 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:38:06.881 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:38:06.881 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:06.884 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.880+0000 7fc21bc6d640 1 Processor -- start 2024-03-29T11:38:06.884 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.881+0000 7fc21bc6d640 1 -- start start 2024-03-29T11:38:06.884 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.881+0000 7fc21bc6d640 10 monclient: init 2024-03-29T11:38:06.884 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.881+0000 7fc21bc6d640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:06.884 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.881+0000 7fc21bc6d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:38:06.884 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.881+0000 7fc21bc6d640 10 monclient(hunting): picked mon.noname-b con 0x7fc2140a9930 addr v1:172.21.15.72:6790/0 2024-03-29T11:38:06.884 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.881+0000 7fc21bc6d640 10 monclient(hunting): picked mon.noname-c con 0x7fc214149dc0 addr v1:172.21.15.89:6789/0 2024-03-29T11:38:06.885 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.881+0000 7fc21bc6d640 10 monclient(hunting): picked mon.noname-a con 0x7fc21414ff40 addr v1:172.21.15.72:6789/0 2024-03-29T11:38:06.885 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.881+0000 7fc21bc6d640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2140aae60 con 0x7fc21414ff40 2024-03-29T11:38:06.885 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.881+0000 7fc21bc6d640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2140b4310 con 0x7fc214149dc0 2024-03-29T11:38:06.885 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.881+0000 7fc21bc6d640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2140afdd0 con 0x7fc2140a9930 2024-03-29T11:38:06.885 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.881+0000 7fc21bc6d640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:06.885 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.881+0000 7fc21bc6d640 10 monclient(hunting): authenticate will time out at 1046.169984s 2024-03-29T11:38:06.886 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.886+0000 7fc21a20a640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fc21414ff40 0x7fc214152330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:53579/0 (socket says 172.21.15.72:53579) 2024-03-29T11:38:06.889 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.886+0000 7fc21a20a640 1 -- 172.21.15.72:0/4262850811 learned_addr learned my addr 172.21.15.72:0/4262850811 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:38:06.889 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.888+0000 7fc218a07640 1 -- 172.21.15.72:0/4262850811 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 114419765 0 0) 0x7fc2140b4310 con 0x7fc214149dc0 2024-03-29T11:38:06.890 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.888+0000 7fc218a07640 10 monclient(hunting): _negotiate 2024-03-29T11:38:06.890 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.888+0000 7fc218a07640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:06.890 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.888+0000 7fc218a07640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:06.890 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.888+0000 7fc218a07640 10 monclient(hunting): my global_id is 4327 2024-03-29T11:38:06.890 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.888+0000 7fc218a07640 1 -- 172.21.15.72:0/4262850811 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc200003520 con 0x7fc214149dc0 2024-03-29T11:38:06.890 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.888+0000 7fc218a07640 1 -- 172.21.15.72:0/4262850811 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3662850970 0 0) 0x7fc2140afdd0 con 0x7fc2140a9930 2024-03-29T11:38:06.890 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.888+0000 7fc218a07640 10 monclient(hunting): _negotiate 2024-03-29T11:38:06.890 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.888+0000 7fc218a07640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:06.890 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.888+0000 7fc218a07640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:06.890 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.888+0000 7fc218a07640 10 monclient(hunting): my global_id is 4337 2024-03-29T11:38:06.890 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.888+0000 7fc218a07640 1 -- 172.21.15.72:0/4262850811 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2140b4310 con 0x7fc2140a9930 2024-03-29T11:38:06.890 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.888+0000 7fc218a07640 1 -- 172.21.15.72:0/4262850811 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2249068395 0 0) 0x7fc2140aae60 con 0x7fc21414ff40 2024-03-29T11:38:06.890 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.888+0000 7fc218a07640 10 monclient(hunting): _negotiate 2024-03-29T11:38:06.890 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.888+0000 7fc218a07640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:06.891 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.888+0000 7fc218a07640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:06.891 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.888+0000 7fc218a07640 10 monclient(hunting): my global_id is 4335 2024-03-29T11:38:06.891 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.888+0000 7fc218a07640 1 -- 172.21.15.72:0/4262850811 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2140afdd0 con 0x7fc21414ff40 2024-03-29T11:38:06.891 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.888+0000 7fc218a07640 1 -- 172.21.15.72:0/4262850811 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 56303643 0 0) 0x7fc200003520 con 0x7fc214149dc0 2024-03-29T11:38:06.891 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.888+0000 7fc218a07640 1 -- 172.21.15.72:0/4262850811 >> v1:172.21.15.72:6790/0 conn(0x7fc2140a9930 legacy=0x7fc214143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:06.891 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.888+0000 7fc218a07640 1 -- 172.21.15.72:0/4262850811 >> v1:172.21.15.72:6789/0 conn(0x7fc21414ff40 legacy=0x7fc214152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:06.891 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.888+0000 7fc218a07640 10 monclient: _finish_hunting 0 2024-03-29T11:38:06.891 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.888+0000 7fc218a07640 1 monclient: found mon.noname-c 2024-03-29T11:38:06.891 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.888+0000 7fc218a07640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:06.891 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.888+0000 7fc218a07640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.89:6789/0 2024-03-29T11:38:06.891 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.888+0000 7fc218a07640 1 -- 172.21.15.72:0/4262850811 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc21414a950 con 0x7fc214149dc0 2024-03-29T11:38:06.891 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.888+0000 7fc218a07640 10 monclient: _finish_auth 0 2024-03-29T11:38:06.891 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.888+0000 7fc218a07640 10 monclient: _check_auth_tickets 2024-03-29T11:38:06.891 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.888+0000 7fc218a07640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:06.891 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.888+0000 7fc21bc6d640 5 monclient: authenticate success, global_id 4327 2024-03-29T11:38:06.892 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.888+0000 7fc21bc6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:06.892 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.888+0000 7fc218a07640 1 -- 172.21.15.72:0/4262850811 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fc2040015c0 con 0x7fc214149dc0 2024-03-29T11:38:06.892 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.888+0000 7fc218a07640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:06.892 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.888+0000 7fc218a07640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:06.892 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.888+0000 7fc218a07640 10 monclient: dump: 2024-03-29T11:38:06.892 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:38:06.892 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:38:06.892 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:06.892 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:06.892 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:06.892 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:06.892 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:38:06.892 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:38:06.892 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:38:06.892 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:06.892 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.888+0000 7fc21bc6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:06.893 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.888+0000 7fc218a07640 1 -- 172.21.15.72:0/4262850811 <== mon.1 v1:172.21.15.89:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2040017e0 con 0x7fc214149dc0 2024-03-29T11:38:06.893 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.888+0000 7fc218a07640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:06.893 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.888+0000 7fc21bc6d640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:06.893 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.888+0000 7fc21bc6d640 4 set_mon_vals no callback set 2024-03-29T11:38:06.893 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.889+0000 7fc21bc6d640 10 monclient: shutdown 2024-03-29T11:38:06.893 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.889+0000 7fc21bc6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:06.893 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.889+0000 7fc21bc6d640 1 -- 172.21.15.72:0/4262850811 >> v1:172.21.15.89:6789/0 conn(0x7fc214149dc0 legacy=0x7fc21414a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:06.894 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.891+0000 7fc21bc6d640 1 -- 172.21.15.72:0/4262850811 shutdown_connections 2024-03-29T11:38:06.894 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.891+0000 7fc21bc6d640 1 -- 172.21.15.72:0/4262850811 >> 172.21.15.72:0/4262850811 conn(0x7fc2140b35e0 msgr2=0x7fc2140b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:06.894 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.891+0000 7fc21bc6d640 1 -- 172.21.15.72:0/4262850811 shutdown_connections 2024-03-29T11:38:06.894 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.891+0000 7fc21bc6d640 1 -- 172.21.15.72:0/4262850811 wait complete. 2024-03-29T11:38:06.898 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.892+0000 7fc21bc6d640 10 monclient: build_initial_monmap 2024-03-29T11:38:06.898 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.892+0000 7fc21bc6d640 10 monclient: monmap: 2024-03-29T11:38:06.898 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:38:06.899 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:06.899 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:38:06.899 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:38:06.899 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:06.899 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:06.899 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:38:06.899 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:38:06.899 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:38:06.899 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:06.899 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.893+0000 7fc21bc6d640 1 librados: starting msgr at 2024-03-29T11:38:06.899 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.893+0000 7fc21bc6d640 1 librados: starting objecter 2024-03-29T11:38:06.899 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.893+0000 7fc21bc6d640 1 Processor -- start 2024-03-29T11:38:06.899 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.893+0000 7fc21bc6d640 1 -- start start 2024-03-29T11:38:06.899 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.893+0000 7fc21bc6d640 1 librados: setting wanted keys 2024-03-29T11:38:06.899 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.893+0000 7fc21bc6d640 1 librados: calling monclient init 2024-03-29T11:38:06.899 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.893+0000 7fc21bc6d640 10 monclient: init 2024-03-29T11:38:06.900 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.894+0000 7fc21bc6d640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:06.900 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.894+0000 7fc21bc6d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:38:06.900 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.895+0000 7fc21bc6d640 10 monclient(hunting): picked mon.noname-b con 0x7fc2140a9930 addr v1:172.21.15.89:6789/0 2024-03-29T11:38:06.900 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.898+0000 7fc21bc6d640 10 monclient(hunting): picked mon.noname-a con 0x7fc214149dc0 addr v1:172.21.15.72:6789/0 2024-03-29T11:38:06.900 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.898+0000 7fc21bc6d640 10 monclient(hunting): picked mon.noname-c con 0x7fc21414ff40 addr v1:172.21.15.72:6790/0 2024-03-29T11:38:06.900 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.898+0000 7fc21bc6d640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc21413be10 con 0x7fc214149dc0 2024-03-29T11:38:06.900 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.898+0000 7fc21bc6d640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc21413c0f0 con 0x7fc2140a9930 2024-03-29T11:38:06.900 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.898+0000 7fc21bc6d640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2141382d0 con 0x7fc21414ff40 2024-03-29T11:38:06.900 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.898+0000 7fc21bc6d640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:06.900 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.898+0000 7fc21bc6d640 10 monclient(hunting): authenticate will time out at 1046.186625s 2024-03-29T11:38:06.902 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.901+0000 7fc219a09640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc2140a9930 0x7fc214142610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:51041/0 (socket says 172.21.15.72:51041) 2024-03-29T11:38:06.902 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.901+0000 7fc219a09640 1 -- 172.21.15.72:0/961535445 learned_addr learned my addr 172.21.15.72:0/961535445 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:38:06.902 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.902+0000 7fc20a7fc640 10 client.?.objecter ms_handle_connect 0x7fc2140a9930 2024-03-29T11:38:06.902 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.902+0000 7fc20a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:06.903 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.902+0000 7fc20a7fc640 1 -- 172.21.15.72:0/961535445 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1273568324 0 0) 0x7fc21413c0f0 con 0x7fc2140a9930 2024-03-29T11:38:06.903 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.903+0000 7fc20a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:38:06.903 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.903+0000 7fc20a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:06.903 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.903+0000 7fc20a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:06.903 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.903+0000 7fc20a7fc640 10 monclient(hunting): my global_id is 4330 2024-03-29T11:38:06.903 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.903+0000 7fc20a7fc640 1 -- 172.21.15.72:0/961535445 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1f8004eb0 con 0x7fc2140a9930 2024-03-29T11:38:06.903 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.903+0000 7fc20a7fc640 1 -- 172.21.15.72:0/961535445 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3755562157 0 0) 0x7fc1f8004eb0 con 0x7fc2140a9930 2024-03-29T11:38:06.903 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.903+0000 7fc20a7fc640 1 -- 172.21.15.72:0/961535445 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc21413c0f0 con 0x7fc2140a9930 2024-03-29T11:38:06.904 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.903+0000 7fc20a7fc640 1 -- 172.21.15.72:0/961535445 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fc210001a30 con 0x7fc2140a9930 2024-03-29T11:38:06.904 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.903+0000 7fc20a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:06.904 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.903+0000 7fc20a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:06.904 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.903+0000 7fc20a7fc640 10 monclient(hunting): dump: 2024-03-29T11:38:06.904 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:38:06.904 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:38:06.904 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:06.904 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:06.904 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:06.904 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:06.904 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:38:06.904 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:38:06.904 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:38:06.904 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:06.905 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.903+0000 7fc20a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:06.907 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.906+0000 7fc20a7fc640 1 -- 172.21.15.72:0/961535445 <== mon.1 v1:172.21.15.89:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 952174660 0 0) 0x7fc21413c0f0 con 0x7fc2140a9930 2024-03-29T11:38:06.907 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.906+0000 7fc20a7fc640 1 -- 172.21.15.72:0/961535445 >> v1:172.21.15.72:6790/0 conn(0x7fc21414ff40 legacy=0x7fc2142000a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:06.907 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.906+0000 7fc20a7fc640 1 -- 172.21.15.72:0/961535445 >> v1:172.21.15.72:6789/0 conn(0x7fc214149dc0 legacy=0x7fc2140ab1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:06.907 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.906+0000 7fc20a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:38:06.908 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.906+0000 7fc20a7fc640 1 monclient: found mon.b 2024-03-29T11:38:06.908 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.906+0000 7fc20a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:06.908 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.906+0000 7fc20a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:38:06.908 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.906+0000 7fc20a7fc640 1 -- 172.21.15.72:0/961535445 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc21413db80 con 0x7fc2140a9930 2024-03-29T11:38:06.908 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.906+0000 7fc20a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:38:06.908 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.906+0000 7fc20a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:38:06.908 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.906+0000 7fc20a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:06.908 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.906+0000 7fc21bc6d640 5 monclient: authenticate success, global_id 4330 2024-03-29T11:38:06.908 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.906+0000 7fc21bc6d640 10 monclient: _renew_subs 2024-03-29T11:38:06.908 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.906+0000 7fc21bc6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:38:06.908 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.906+0000 7fc21bc6d640 1 -- 172.21.15.72:0/961535445 --> v1:172.21.15.89:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc214143310 con 0x7fc2140a9930 2024-03-29T11:38:06.908 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.906+0000 7fc21bc6d640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:06.908 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.906+0000 7fc21bc6d640 10 monclient: _renew_subs 2024-03-29T11:38:06.908 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.906+0000 7fc21bc6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:38:06.908 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.906+0000 7fc21bc6d640 1 -- 172.21.15.72:0/961535445 --> v1:172.21.15.89:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc214137c00 con 0x7fc2140a9930 2024-03-29T11:38:06.909 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.906+0000 7fc21bc6d640 1 librados: init done 2024-03-29T11:38:06.909 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.907+0000 7fc20a7fc640 1 -- 172.21.15.72:0/961535445 <== mon.1 v1:172.21.15.89:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc210001b90 con 0x7fc2140a9930 2024-03-29T11:38:06.909 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.907+0000 7fc20a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:06.909 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.907+0000 7fc20a7fc640 1 -- 172.21.15.72:0/961535445 <== mon.1 v1:172.21.15.89:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fc210003a60 con 0x7fc2140a9930 2024-03-29T11:38:06.909 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.907+0000 7fc20a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:06.909 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.907+0000 7fc20a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:06.909 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.907+0000 7fc20a7fc640 10 monclient: dump: 2024-03-29T11:38:06.909 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:38:06.909 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:38:06.909 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:06.909 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:06.909 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:06.909 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:06.909 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:38:06.909 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:38:06.909 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:38:06.910 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:06.911 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.907+0000 7fc20b7fe640 4 set_mon_vals no callback set 2024-03-29T11:38:06.911 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.908+0000 7fc21bc6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:06.911 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.908+0000 7fc21bc6d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:06.911 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.908+0000 7fc21bc6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:38:06.911 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.908+0000 7fc21bc6d640 1 -- 172.21.15.72:0/961535445 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1fc005080 con 0x7fc2140a9930 2024-03-29T11:38:06.911 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.909+0000 7fc20a7fc640 1 -- 172.21.15.72:0/961535445 <== mon.1 v1:172.21.15.89:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7fc210003bc0 con 0x7fc2140a9930 2024-03-29T11:38:06.911 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.910+0000 7fc20a7fc640 1 -- 172.21.15.72:0/961535445 <== mon.1 v1:172.21.15.89:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7fc210084c20 con 0x7fc2140a9930 2024-03-29T11:38:06.911 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.910+0000 7fc20a7fc640 10 client.4330.objecter ms_dispatch 0x7fc214000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:06.911 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.910+0000 7fc20a7fc640 3 client.4330.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:06.911 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.910+0000 7fc20a7fc640 3 client.4330.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:06.912 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.910+0000 7fc20a7fc640 20 client.4330.objecter dump_active .. 0 homeless 2024-03-29T11:38:06.912 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.911+0000 7fc219208640 10 monclient: get_auth_request con 0x7fc1f806dc90 auth_method 0 2024-03-29T11:38:06.912 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.911+0000 7fc20a7fc640 10 client.4330.objecter ms_handle_connect 0x7fc1f806dc90 2024-03-29T11:38:06.918 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.917+0000 7fc20a7fc640 1 -- 172.21.15.72:0/961535445 <== mon.1 v1:172.21.15.89:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc210050330 con 0x7fc2140a9930 2024-03-29T11:38:06.918 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.917+0000 7fc20a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:06.918 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:06.917+0000 7fc20a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:07.237 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:07.237+0000 7fc21bc6d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:07.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:07.237+0000 7fc21bc6d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:07.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:07.237+0000 7fc21bc6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:38:07.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:07.237+0000 7fc21bc6d640 1 -- 172.21.15.72:0/961535445 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc1fc005330 con 0x7fc2140a9930 2024-03-29T11:38:07.239 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:07.239+0000 7fc20a7fc640 1 -- 172.21.15.72:0/961535445 <== mon.1 v1:172.21.15.89:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7fc210052430 con 0x7fc2140a9930 2024-03-29T11:38:07.239 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:07.239+0000 7fc20a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:07.239 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:07.239+0000 7fc20a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:07.243 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:38:07.243 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:07.244 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:07.242+0000 7fc21bc6d640 10 librados: watch_flush enter 2024-03-29T11:38:07.244 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:07.243+0000 7fc21bc6d640 10 librados: watch_flush exit 2024-03-29T11:38:07.244 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:07.243+0000 7fc21bc6d640 20 client.4330.objecter shutdown clearing up homeless session... 2024-03-29T11:38:07.244 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:07.243+0000 7fc21bc6d640 10 client.4330.objecter successfully canceled tick 2024-03-29T11:38:07.244 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:07.243+0000 7fc21bc6d640 1 -- 172.21.15.72:0/961535445 >> v1:172.21.15.72:6816/1578866102 conn(0x7fc1f806dc90 legacy=0x7fc1f8070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:07.244 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:07.243+0000 7fc21bc6d640 10 monclient: shutdown 2024-03-29T11:38:07.244 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:07.243+0000 7fc21bc6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:07.244 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:07.243+0000 7fc21bc6d640 1 -- 172.21.15.72:0/961535445 >> v1:172.21.15.89:6789/0 conn(0x7fc2140a9930 legacy=0x7fc214142610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:07.244 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:07.244+0000 7fc21bc6d640 1 -- 172.21.15.72:0/961535445 shutdown_connections 2024-03-29T11:38:07.244 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:07.244+0000 7fc21bc6d640 1 -- 172.21.15.72:0/961535445 >> 172.21.15.72:0/961535445 conn(0x7fc2140b35e0 msgr2=0x7fc2140b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:07.245 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:07.244+0000 7fc21bc6d640 1 -- 172.21.15.72:0/961535445 shutdown_connections 2024-03-29T11:38:07.245 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:07.244+0000 7fc21bc6d640 1 -- 172.21.15.72:0/961535445 wait complete. 2024-03-29T11:38:07.245 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:07.244+0000 7fc21bc6d640 1 librados: shutdown 2024-03-29T11:38:07.261 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:14.069 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:14.217 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.217+0000 7f5edafc9640 10 monclient: get_monmap_and_config 2024-03-29T11:38:14.218 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.217+0000 7f5edafc9640 10 monclient: build_initial_monmap 2024-03-29T11:38:14.218 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.217+0000 7f5edafc9640 10 monclient: monmap: 2024-03-29T11:38:14.218 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:38:14.218 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:38:14.218 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:38:14.218137+0000 2024-03-29T11:38:14.218 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:38:14.218137+0000 2024-03-29T11:38:14.218 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:14.218 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:14.218 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:38:14.218 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:38:14.218 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:38:14.218 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:14.219 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.218+0000 7f5edafc9640 1 Processor -- start 2024-03-29T11:38:14.219 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.218+0000 7f5edafc9640 1 -- start start 2024-03-29T11:38:14.219 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.218+0000 7f5edafc9640 10 monclient: init 2024-03-29T11:38:14.219 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.219+0000 7f5edafc9640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:14.219 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.219+0000 7f5edafc9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:38:14.222 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.219+0000 7f5edafc9640 10 monclient(hunting): picked mon.noname-b con 0x7f5ed40b0770 addr v1:172.21.15.72:6790/0 2024-03-29T11:38:14.222 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.219+0000 7f5edafc9640 10 monclient(hunting): picked mon.noname-c con 0x7f5ed414a700 addr v1:172.21.15.89:6789/0 2024-03-29T11:38:14.223 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.219+0000 7f5edafc9640 10 monclient(hunting): picked mon.noname-a con 0x7f5ed40ac5a0 addr v1:172.21.15.72:6789/0 2024-03-29T11:38:14.223 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.219+0000 7f5edafc9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed40aba70 con 0x7f5ed40ac5a0 2024-03-29T11:38:14.223 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.219+0000 7f5edafc9640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed40aecb0 con 0x7f5ed414a700 2024-03-29T11:38:14.223 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.219+0000 7f5edafc9640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed40af150 con 0x7f5ed40b0770 2024-03-29T11:38:14.223 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.219+0000 7f5edafc9640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:14.223 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.219+0000 7f5edafc9640 10 monclient(hunting): authenticate will time out at 1053.507777s 2024-03-29T11:38:14.223 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.223+0000 7f5ed8d65640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f5ed40b0770 0x7f5ed4149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:54386/0 (socket says 172.21.15.72:54386) 2024-03-29T11:38:14.223 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.223+0000 7f5ed3fff640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f5ed414a700 0x7f5ed40abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:49239/0 (socket says 172.21.15.72:49239) 2024-03-29T11:38:14.224 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.223+0000 7f5ed3fff640 1 -- 172.21.15.72:0/1268006145 learned_addr learned my addr 172.21.15.72:0/1268006145 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:38:14.225 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.225+0000 7f5ed37fe640 1 -- 172.21.15.72:0/1268006145 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1539048244 0 0) 0x7f5ed40aecb0 con 0x7f5ed414a700 2024-03-29T11:38:14.225 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.225+0000 7f5ed37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:38:14.225 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.225+0000 7f5ed37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:14.225 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.225+0000 7f5ed37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:14.225 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.225+0000 7f5ed37fe640 10 monclient(hunting): my global_id is 4333 2024-03-29T11:38:14.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.225+0000 7f5ed37fe640 1 -- 172.21.15.72:0/1268006145 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ec4003520 con 0x7f5ed414a700 2024-03-29T11:38:14.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.225+0000 7f5ed37fe640 1 -- 172.21.15.72:0/1268006145 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 189169945 0 0) 0x7f5ed40aba70 con 0x7f5ed40ac5a0 2024-03-29T11:38:14.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.225+0000 7f5ed37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:38:14.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.225+0000 7f5ed37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:14.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.225+0000 7f5ed37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:14.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.225+0000 7f5ed37fe640 10 monclient(hunting): my global_id is 4338 2024-03-29T11:38:14.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.225+0000 7f5ed37fe640 1 -- 172.21.15.72:0/1268006145 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ed40aecb0 con 0x7f5ed40ac5a0 2024-03-29T11:38:14.227 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.225+0000 7f5ed37fe640 1 -- 172.21.15.72:0/1268006145 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 89202443 0 0) 0x7f5ec4003520 con 0x7f5ed414a700 2024-03-29T11:38:14.227 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.225+0000 7f5ed37fe640 1 -- 172.21.15.72:0/1268006145 >> v1:172.21.15.72:6790/0 conn(0x7f5ed40b0770 legacy=0x7f5ed4149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:14.227 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.225+0000 7f5ed37fe640 1 -- 172.21.15.72:0/1268006145 >> v1:172.21.15.72:6789/0 conn(0x7f5ed40ac5a0 legacy=0x7f5ed4151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:14.227 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.226+0000 7f5ed37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:38:14.227 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.226+0000 7f5ed37fe640 1 monclient: found mon.noname-c 2024-03-29T11:38:14.227 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.226+0000 7f5ed37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:14.227 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.226+0000 7f5ed37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.89:6789/0 2024-03-29T11:38:14.227 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.226+0000 7f5ed37fe640 1 -- 172.21.15.72:0/1268006145 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed40ac9d0 con 0x7f5ed414a700 2024-03-29T11:38:14.227 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.226+0000 7f5ed37fe640 10 monclient: _finish_auth 0 2024-03-29T11:38:14.227 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.226+0000 7f5ed37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:38:14.227 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.226+0000 7f5ed37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:14.227 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.226+0000 7f5ed37fe640 1 -- 172.21.15.72:0/1268006145 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f5ec80016b0 con 0x7f5ed414a700 2024-03-29T11:38:14.227 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.226+0000 7f5ed37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:14.227 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.226+0000 7f5ed37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:14.228 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.226+0000 7f5ed37fe640 10 monclient: dump: 2024-03-29T11:38:14.228 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:38:14.228 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:38:14.228 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:14.228 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:14.228 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:14.228 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:14.228 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:38:14.228 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:38:14.228 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:38:14.228 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:14.228 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.226+0000 7f5edafc9640 5 monclient: authenticate success, global_id 4333 2024-03-29T11:38:14.228 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.226+0000 7f5edafc9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:14.230 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.228+0000 7f5ed37fe640 1 -- 172.21.15.72:0/1268006145 <== mon.1 v1:172.21.15.89:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ec80018d0 con 0x7f5ed414a700 2024-03-29T11:38:14.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.228+0000 7f5ed37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:14.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.228+0000 7f5ed37fe640 1 -- 172.21.15.72:0/1268006145 <== mon.1 v1:172.21.15.89:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f5ec8001c60 con 0x7f5ed414a700 2024-03-29T11:38:14.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.228+0000 7f5ed37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:14.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.228+0000 7f5ed37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:14.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.228+0000 7f5ed37fe640 10 monclient: dump: 2024-03-29T11:38:14.231 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:38:14.231 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:38:14.231 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:14.231 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:14.231 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:14.231 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:14.232 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:38:14.232 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:38:14.232 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:38:14.232 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:14.232 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.230+0000 7f5edafc9640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:14.232 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.230+0000 7f5edafc9640 4 set_mon_vals no callback set 2024-03-29T11:38:14.232 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.232+0000 7f5edafc9640 10 monclient: shutdown 2024-03-29T11:38:14.232 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.232+0000 7f5edafc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:14.233 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.232+0000 7f5edafc9640 1 -- 172.21.15.72:0/1268006145 >> v1:172.21.15.89:6789/0 conn(0x7f5ed414a700 legacy=0x7f5ed40abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:14.233 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.232+0000 7f5edafc9640 1 -- 172.21.15.72:0/1268006145 shutdown_connections 2024-03-29T11:38:14.233 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.232+0000 7f5edafc9640 1 -- 172.21.15.72:0/1268006145 >> 172.21.15.72:0/1268006145 conn(0x7f5ed406b660 msgr2=0x7f5ed406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:14.236 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.235+0000 7f5edafc9640 1 -- 172.21.15.72:0/1268006145 shutdown_connections 2024-03-29T11:38:14.236 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.235+0000 7f5edafc9640 1 -- 172.21.15.72:0/1268006145 wait complete. 2024-03-29T11:38:14.237 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.236+0000 7f5edafc9640 10 monclient: build_initial_monmap 2024-03-29T11:38:14.237 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.236+0000 7f5edafc9640 10 monclient: monmap: 2024-03-29T11:38:14.237 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:38:14.237 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:14.238 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:38:14.238 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:38:14.238 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:14.238 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:14.238 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:38:14.238 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:38:14.238 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:38:14.238 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:14.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.237+0000 7f5edafc9640 1 librados: starting msgr at 2024-03-29T11:38:14.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.237+0000 7f5edafc9640 1 librados: starting objecter 2024-03-29T11:38:14.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.237+0000 7f5edafc9640 1 Processor -- start 2024-03-29T11:38:14.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.237+0000 7f5edafc9640 1 -- start start 2024-03-29T11:38:14.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.237+0000 7f5edafc9640 1 librados: setting wanted keys 2024-03-29T11:38:14.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.237+0000 7f5edafc9640 1 librados: calling monclient init 2024-03-29T11:38:14.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.237+0000 7f5edafc9640 10 monclient: init 2024-03-29T11:38:14.240 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.238+0000 7f5edafc9640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:14.240 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.238+0000 7f5edafc9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:38:14.240 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.238+0000 7f5edafc9640 10 monclient(hunting): picked mon.noname-b con 0x7f5ed40ac5a0 addr v1:172.21.15.89:6789/0 2024-03-29T11:38:14.240 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.239+0000 7f5edafc9640 10 monclient(hunting): picked mon.noname-c con 0x7f5ed40b0770 addr v1:172.21.15.72:6790/0 2024-03-29T11:38:14.241 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.239+0000 7f5ed3fff640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f5ed40b0770 0x7f5ed4144ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:43442/0 (socket says 172.21.15.72:43442) 2024-03-29T11:38:14.241 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.239+0000 7f5ed3fff640 1 -- 172.21.15.72:0/3159358746 learned_addr learned my addr 172.21.15.72:0/3159358746 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:38:14.241 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.239+0000 7f5ed17fa640 10 client.?.objecter ms_handle_connect 0x7f5ed40b0770 2024-03-29T11:38:14.241 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.239+0000 7f5ed17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:14.241 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.239+0000 7f5edafc9640 10 monclient(hunting): picked mon.noname-a con 0x7f5ed4071330 addr v1:172.21.15.72:6789/0 2024-03-29T11:38:14.241 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.239+0000 7f5edafc9640 1 -- 172.21.15.72:0/3159358746 --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed41e4190 con 0x7f5ed4071330 2024-03-29T11:38:14.241 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.239+0000 7f5edafc9640 1 -- 172.21.15.72:0/3159358746 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed4145900 con 0x7f5ed40ac5a0 2024-03-29T11:38:14.241 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.239+0000 7f5edafc9640 1 -- 172.21.15.72:0/3159358746 --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed411bcb0 con 0x7f5ed40b0770 2024-03-29T11:38:14.241 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.239+0000 7f5edafc9640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:14.241 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.239+0000 7f5edafc9640 10 monclient(hunting): authenticate will time out at 1053.527896s 2024-03-29T11:38:14.247 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.240+0000 7f5ed17fa640 1 -- 172.21.15.72:0/3159358746 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3443835194 0 0) 0x7f5ed411bcb0 con 0x7f5ed40b0770 2024-03-29T11:38:14.247 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.240+0000 7f5ed17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:38:14.247 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.240+0000 7f5ed17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:14.248 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.240+0000 7f5ed17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:14.248 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.240+0000 7f5ed17fa640 10 monclient(hunting): my global_id is 4340 2024-03-29T11:38:14.248 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.240+0000 7f5ed17fa640 1 -- 172.21.15.72:0/3159358746 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5eb8004eb0 con 0x7f5ed40b0770 2024-03-29T11:38:14.248 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.240+0000 7f5ed17fa640 1 -- 172.21.15.72:0/3159358746 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3686392659 0 0) 0x7f5eb8004eb0 con 0x7f5ed40b0770 2024-03-29T11:38:14.248 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.240+0000 7f5ed17fa640 1 -- 172.21.15.72:0/3159358746 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ed411bcb0 con 0x7f5ed40b0770 2024-03-29T11:38:14.248 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.240+0000 7f5ed17fa640 1 -- 172.21.15.72:0/3159358746 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f5ec80016b0 con 0x7f5ed40b0770 2024-03-29T11:38:14.248 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.240+0000 7f5ed17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:14.248 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.240+0000 7f5ed17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:14.248 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.240+0000 7f5ed17fa640 10 monclient(hunting): dump: 2024-03-29T11:38:14.248 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:38:14.248 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:38:14.248 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:14.248 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:14.248 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:14.248 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:14.249 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:38:14.249 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:38:14.249 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:38:14.249 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:14.249 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.240+0000 7f5ed17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:14.249 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.241+0000 7f5ed17fa640 1 -- 172.21.15.72:0/3159358746 <== mon.2 v1:172.21.15.72:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1465068711 0 0) 0x7f5ed411bcb0 con 0x7f5ed40b0770 2024-03-29T11:38:14.249 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.241+0000 7f5ed17fa640 1 -- 172.21.15.72:0/3159358746 >> v1:172.21.15.89:6789/0 conn(0x7f5ed40ac5a0 legacy=0x7f5ed41e4620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:38:14.249 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.244+0000 7f5ed17fa640 1 -- 172.21.15.72:0/3159358746 >> v1:172.21.15.72:6789/0 conn(0x7f5ed4071330 legacy=0x7f5ed41451f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:14.249 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.244+0000 7f5ed17fa640 10 monclient: _finish_hunting 0 2024-03-29T11:38:14.249 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.244+0000 7f5ed17fa640 1 monclient: found mon.c 2024-03-29T11:38:14.249 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.244+0000 7f5ed17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:14.249 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.244+0000 7f5ed17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:38:14.249 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.244+0000 7f5ed17fa640 1 -- 172.21.15.72:0/3159358746 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed4070410 con 0x7f5ed40b0770 2024-03-29T11:38:14.249 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.244+0000 7f5ed17fa640 10 monclient: _finish_auth 0 2024-03-29T11:38:14.249 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.244+0000 7f5ed17fa640 10 monclient: _check_auth_tickets 2024-03-29T11:38:14.250 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.244+0000 7f5ed17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:14.250 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.244+0000 7f5edafc9640 5 monclient: authenticate success, global_id 4340 2024-03-29T11:38:14.250 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.245+0000 7f5edafc9640 10 monclient: _renew_subs 2024-03-29T11:38:14.250 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.245+0000 7f5edafc9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:38:14.250 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.245+0000 7f5edafc9640 1 -- 172.21.15.72:0/3159358746 --> v1:172.21.15.72:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ed4071900 con 0x7f5ed40b0770 2024-03-29T11:38:14.250 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.245+0000 7f5edafc9640 10 client.4340.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:14.250 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.245+0000 7f5edafc9640 10 monclient: _renew_subs 2024-03-29T11:38:14.250 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.245+0000 7f5edafc9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:38:14.250 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.245+0000 7f5edafc9640 1 -- 172.21.15.72:0/3159358746 --> v1:172.21.15.72:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ed406de10 con 0x7f5ed40b0770 2024-03-29T11:38:14.250 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.245+0000 7f5edafc9640 1 librados: init done 2024-03-29T11:38:14.250 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.245+0000 7f5ed17fa640 1 -- 172.21.15.72:0/3159358746 <== mon.2 v1:172.21.15.72:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ec8004a90 con 0x7f5ed40b0770 2024-03-29T11:38:14.250 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.246+0000 7f5ed17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:14.250 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.246+0000 7f5ed17fa640 1 -- 172.21.15.72:0/3159358746 <== mon.2 v1:172.21.15.72:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f5ec8004ea0 con 0x7f5ed40b0770 2024-03-29T11:38:14.250 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.246+0000 7f5ed17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:14.250 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.246+0000 7f5ed17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:14.250 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.246+0000 7f5ed17fa640 10 monclient: dump: 2024-03-29T11:38:14.251 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:38:14.251 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:38:14.251 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:14.251 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:14.251 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:14.251 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:14.251 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:38:14.251 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:38:14.251 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:38:14.251 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:14.251 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.246+0000 7f5ed27fc640 4 set_mon_vals no callback set 2024-03-29T11:38:14.252 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.248+0000 7f5ed17fa640 1 -- 172.21.15.72:0/3159358746 <== mon.2 v1:172.21.15.72:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f5ec8005000 con 0x7f5ed40b0770 2024-03-29T11:38:14.253 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.248+0000 7f5ed17fa640 1 -- 172.21.15.72:0/3159358746 <== mon.2 v1:172.21.15.72:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f5ec8084c90 con 0x7f5ed40b0770 2024-03-29T11:38:14.253 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.248+0000 7f5ed17fa640 10 client.4340.objecter ms_dispatch 0x7f5ed4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:14.253 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.248+0000 7f5ed17fa640 3 client.4340.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:14.253 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.248+0000 7f5ed17fa640 3 client.4340.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:14.253 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.249+0000 7f5ed17fa640 20 client.4340.objecter dump_active .. 0 homeless 2024-03-29T11:38:14.253 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.249+0000 7f5ed8d65640 10 monclient: get_auth_request con 0x7f5eb806dcd0 auth_method 0 2024-03-29T11:38:14.253 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.249+0000 7f5edafc9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:14.253 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.249+0000 7f5ed17fa640 10 client.4340.objecter ms_handle_connect 0x7f5eb806dcd0 2024-03-29T11:38:14.253 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.250+0000 7f5edafc9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:14.253 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.250+0000 7f5edafc9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:38:14.253 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.250+0000 7f5edafc9640 1 -- 172.21.15.72:0/3159358746 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5eb4005080 con 0x7f5ed40b0770 2024-03-29T11:38:14.259 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.258+0000 7f5ed17fa640 1 -- 172.21.15.72:0/3159358746 <== mon.2 v1:172.21.15.72:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5ec8050570 con 0x7f5ed40b0770 2024-03-29T11:38:14.259 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.258+0000 7f5ed17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:14.259 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.258+0000 7f5ed17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:14.579 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.579+0000 7f5edafc9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:14.579 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.579+0000 7f5edafc9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:14.579 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.579+0000 7f5edafc9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:38:14.580 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.579+0000 7f5edafc9640 1 -- 172.21.15.72:0/3159358746 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5eb4005330 con 0x7f5ed40b0770 2024-03-29T11:38:14.581 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.581+0000 7f5ed17fa640 1 -- 172.21.15.72:0/3159358746 <== mon.2 v1:172.21.15.72:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f5ec80524a0 con 0x7f5ed40b0770 2024-03-29T11:38:14.581 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.581+0000 7f5ed17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:14.581 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.581+0000 7f5ed17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:14.582 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:38:14.582 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:14.588 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.587+0000 7f5edafc9640 10 librados: watch_flush enter 2024-03-29T11:38:14.588 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.587+0000 7f5edafc9640 10 librados: watch_flush exit 2024-03-29T11:38:14.588 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.587+0000 7f5edafc9640 20 client.4340.objecter shutdown clearing up homeless session... 2024-03-29T11:38:14.588 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.587+0000 7f5edafc9640 10 client.4340.objecter successfully canceled tick 2024-03-29T11:38:14.588 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.588+0000 7f5edafc9640 1 -- 172.21.15.72:0/3159358746 >> v1:172.21.15.72:6816/1578866102 conn(0x7f5eb806dcd0 legacy=0x7f5eb8070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:14.588 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.588+0000 7f5edafc9640 10 monclient: shutdown 2024-03-29T11:38:14.588 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.588+0000 7f5edafc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:14.588 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.588+0000 7f5edafc9640 1 -- 172.21.15.72:0/3159358746 >> v1:172.21.15.72:6790/0 conn(0x7f5ed40b0770 legacy=0x7f5ed4144ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:14.592 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.592+0000 7f5edafc9640 1 -- 172.21.15.72:0/3159358746 shutdown_connections 2024-03-29T11:38:14.592 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.592+0000 7f5edafc9640 1 -- 172.21.15.72:0/3159358746 >> 172.21.15.72:0/3159358746 conn(0x7f5ed406b660 msgr2=0x7f5ed40b3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:14.592 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.592+0000 7f5edafc9640 1 -- 172.21.15.72:0/3159358746 shutdown_connections 2024-03-29T11:38:14.592 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.592+0000 7f5edafc9640 1 -- 172.21.15.72:0/3159358746 wait complete. 2024-03-29T11:38:14.592 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:14.592+0000 7f5edafc9640 1 librados: shutdown 2024-03-29T11:38:14.611 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:21.419 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:21.555 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.554+0000 7f3c38ce0640 10 monclient: get_monmap_and_config 2024-03-29T11:38:21.555 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.554+0000 7f3c38ce0640 10 monclient: build_initial_monmap 2024-03-29T11:38:21.555 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.554+0000 7f3c38ce0640 10 monclient: monmap: 2024-03-29T11:38:21.555 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:38:21.555 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:38:21.555 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:38:21.555166+0000 2024-03-29T11:38:21.556 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:38:21.555166+0000 2024-03-29T11:38:21.556 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:21.556 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:21.556 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:38:21.556 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:38:21.556 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:38:21.556 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:21.556 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.555+0000 7f3c38ce0640 1 Processor -- start 2024-03-29T11:38:21.556 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.555+0000 7f3c38ce0640 1 -- start start 2024-03-29T11:38:21.557 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.555+0000 7f3c38ce0640 10 monclient: init 2024-03-29T11:38:21.559 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.556+0000 7f3c38ce0640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:21.559 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.556+0000 7f3c38ce0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:38:21.559 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.556+0000 7f3c38ce0640 10 monclient(hunting): picked mon.noname-a con 0x7f3c340a9930 addr v1:172.21.15.72:6789/0 2024-03-29T11:38:21.559 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.556+0000 7f3c38ce0640 10 monclient(hunting): picked mon.noname-c con 0x7f3c34149dc0 addr v1:172.21.15.89:6789/0 2024-03-29T11:38:21.559 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.556+0000 7f3c38ce0640 10 monclient(hunting): picked mon.noname-b con 0x7f3c3414ff40 addr v1:172.21.15.72:6790/0 2024-03-29T11:38:21.559 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.556+0000 7f3c38ce0640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c340aae60 con 0x7f3c340a9930 2024-03-29T11:38:21.559 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.556+0000 7f3c38ce0640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c340b4310 con 0x7f3c34149dc0 2024-03-29T11:38:21.559 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.556+0000 7f3c38ce0640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c340afdd0 con 0x7f3c3414ff40 2024-03-29T11:38:21.559 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.556+0000 7f3c38ce0640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:21.560 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.556+0000 7f3c38ce0640 10 monclient(hunting): authenticate will time out at 1060.844279s 2024-03-29T11:38:21.560 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.556+0000 7f3c32d9d640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f3c3414ff40 0x7f3c34152330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:54782/0 (socket says 172.21.15.72:54782) 2024-03-29T11:38:21.560 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.556+0000 7f3c32d9d640 1 -- 172.21.15.72:0/111991023 learned_addr learned my addr 172.21.15.72:0/111991023 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:38:21.560 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.557+0000 7f3c3159a640 1 -- 172.21.15.72:0/111991023 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 205344947 0 0) 0x7f3c340afdd0 con 0x7f3c3414ff40 2024-03-29T11:38:21.560 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.557+0000 7f3c3159a640 10 monclient(hunting): _negotiate 2024-03-29T11:38:21.560 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.557+0000 7f3c3159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:21.560 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.557+0000 7f3c3159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:21.560 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.557+0000 7f3c3159a640 10 monclient(hunting): my global_id is 4343 2024-03-29T11:38:21.560 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.557+0000 7f3c3159a640 1 -- 172.21.15.72:0/111991023 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c1c003520 con 0x7f3c3414ff40 2024-03-29T11:38:21.560 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.557+0000 7f3c3159a640 1 -- 172.21.15.72:0/111991023 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4147165630 0 0) 0x7f3c340aae60 con 0x7f3c340a9930 2024-03-29T11:38:21.560 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.557+0000 7f3c3159a640 10 monclient(hunting): _negotiate 2024-03-29T11:38:21.560 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.557+0000 7f3c3159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:21.561 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.557+0000 7f3c3159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:21.561 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.557+0000 7f3c3159a640 10 monclient(hunting): my global_id is 4341 2024-03-29T11:38:21.561 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.557+0000 7f3c3159a640 1 -- 172.21.15.72:0/111991023 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c340afdd0 con 0x7f3c340a9930 2024-03-29T11:38:21.561 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.557+0000 7f3c3159a640 1 -- 172.21.15.72:0/111991023 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3847920345 0 0) 0x7f3c340afdd0 con 0x7f3c340a9930 2024-03-29T11:38:21.561 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.558+0000 7f3c3159a640 1 -- 172.21.15.72:0/111991023 >> v1:172.21.15.72:6790/0 conn(0x7f3c3414ff40 legacy=0x7f3c34152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:21.561 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.558+0000 7f3c3159a640 1 -- 172.21.15.72:0/111991023 >> v1:172.21.15.89:6789/0 conn(0x7f3c34149dc0 legacy=0x7f3c3414a240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:38:21.567 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.560+0000 7f3c3159a640 10 monclient: _finish_hunting 0 2024-03-29T11:38:21.567 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.560+0000 7f3c3159a640 1 monclient: found mon.noname-a 2024-03-29T11:38:21.567 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.560+0000 7f3c3159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:21.567 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.560+0000 7f3c3159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.72:6789/0 2024-03-29T11:38:21.568 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.560+0000 7f3c3159a640 1 -- 172.21.15.72:0/111991023 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c3414a950 con 0x7f3c340a9930 2024-03-29T11:38:21.568 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.560+0000 7f3c3159a640 10 monclient: _finish_auth 0 2024-03-29T11:38:21.568 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.560+0000 7f3c3159a640 10 monclient: _check_auth_tickets 2024-03-29T11:38:21.568 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.560+0000 7f3c3159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:21.568 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.560+0000 7f3c38ce0640 5 monclient: authenticate success, global_id 4341 2024-03-29T11:38:21.568 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.560+0000 7f3c38ce0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:21.568 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.560+0000 7f3c3159a640 1 -- 172.21.15.72:0/111991023 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f3c280015f0 con 0x7f3c340a9930 2024-03-29T11:38:21.568 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.560+0000 7f3c3159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:21.568 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.560+0000 7f3c3159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:38:21.569 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.560+0000 7f3c3159a640 10 monclient: dump: 2024-03-29T11:38:21.569 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:38:21.569 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:38:21.569 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:21.569 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:21.569 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:21.569 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:21.569 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:38:21.569 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:38:21.569 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:38:21.569 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:21.569 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.560+0000 7f3c38ce0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:21.570 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.560+0000 7f3c3159a640 1 -- 172.21.15.72:0/111991023 <== mon.0 v1:172.21.15.72:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c28001810 con 0x7f3c340a9930 2024-03-29T11:38:21.570 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.560+0000 7f3c3159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:21.570 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.560+0000 7f3c38ce0640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:21.570 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.560+0000 7f3c38ce0640 4 set_mon_vals no callback set 2024-03-29T11:38:21.570 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.560+0000 7f3c3159a640 1 -- 172.21.15.72:0/111991023 <== mon.0 v1:172.21.15.72:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f3c28001ba0 con 0x7f3c340a9930 2024-03-29T11:38:21.570 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.560+0000 7f3c3159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:21.570 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.560+0000 7f3c3159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:21.570 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.560+0000 7f3c3159a640 10 monclient: dump: 2024-03-29T11:38:21.570 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:38:21.570 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:38:21.570 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:21.571 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:21.571 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:21.571 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:21.571 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:38:21.571 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:38:21.571 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:38:21.571 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:21.571 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.561+0000 7f3c38ce0640 10 monclient: shutdown 2024-03-29T11:38:21.571 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.561+0000 7f3c38ce0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:21.571 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.561+0000 7f3c38ce0640 1 -- 172.21.15.72:0/111991023 >> v1:172.21.15.72:6789/0 conn(0x7f3c340a9930 legacy=0x7f3c34143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:21.571 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.561+0000 7f3c38ce0640 1 -- 172.21.15.72:0/111991023 shutdown_connections 2024-03-29T11:38:21.571 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.561+0000 7f3c38ce0640 1 -- 172.21.15.72:0/111991023 >> 172.21.15.72:0/111991023 conn(0x7f3c340b35e0 msgr2=0x7f3c340b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:21.571 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.562+0000 7f3c38ce0640 1 -- 172.21.15.72:0/111991023 shutdown_connections 2024-03-29T11:38:21.572 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.562+0000 7f3c38ce0640 1 -- 172.21.15.72:0/111991023 wait complete. 2024-03-29T11:38:21.572 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.563+0000 7f3c38ce0640 10 monclient: build_initial_monmap 2024-03-29T11:38:21.572 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.563+0000 7f3c38ce0640 10 monclient: monmap: 2024-03-29T11:38:21.572 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:38:21.572 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:21.572 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:38:21.572 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:38:21.572 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:21.572 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:21.572 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:38:21.572 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:38:21.572 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:38:21.572 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:21.572 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.563+0000 7f3c38ce0640 1 librados: starting msgr at 2024-03-29T11:38:21.573 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.563+0000 7f3c38ce0640 1 librados: starting objecter 2024-03-29T11:38:21.573 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.563+0000 7f3c38ce0640 1 Processor -- start 2024-03-29T11:38:21.573 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.563+0000 7f3c38ce0640 1 -- start start 2024-03-29T11:38:21.573 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.563+0000 7f3c38ce0640 1 librados: setting wanted keys 2024-03-29T11:38:21.573 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.563+0000 7f3c38ce0640 1 librados: calling monclient init 2024-03-29T11:38:21.573 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.563+0000 7f3c38ce0640 10 monclient: init 2024-03-29T11:38:21.573 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.564+0000 7f3c38ce0640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:21.573 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.564+0000 7f3c38ce0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:38:21.573 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.564+0000 7f3c38ce0640 10 monclient(hunting): picked mon.noname-c con 0x7f3c340a9930 addr v1:172.21.15.72:6790/0 2024-03-29T11:38:21.573 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.564+0000 7f3c38ce0640 10 monclient(hunting): picked mon.noname-b con 0x7f3c34149dc0 addr v1:172.21.15.89:6789/0 2024-03-29T11:38:21.573 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.564+0000 7f3c38ce0640 10 monclient(hunting): picked mon.noname-a con 0x7f3c3414ff40 addr v1:172.21.15.72:6789/0 2024-03-29T11:38:21.573 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.564+0000 7f3c38ce0640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c34144fe0 con 0x7f3c3414ff40 2024-03-29T11:38:21.573 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.564+0000 7f3c38ce0640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c341db700 con 0x7f3c34149dc0 2024-03-29T11:38:21.574 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.564+0000 7f3c38ce0640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c34147e50 con 0x7f3c340a9930 2024-03-29T11:38:21.574 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.564+0000 7f3c38ce0640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:21.574 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.564+0000 7f3c38ce0640 10 monclient(hunting): authenticate will time out at 1060.852316s 2024-03-29T11:38:21.574 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.565+0000 7f3c32d9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f3c3414ff40 0x7f3c341e6e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:36982/0 (socket says 172.21.15.72:36982) 2024-03-29T11:38:21.574 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.565+0000 7f3c32d9d640 1 -- 172.21.15.72:0/1085887425 learned_addr learned my addr 172.21.15.72:0/1085887425 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:38:21.574 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.565+0000 7f3c22ffd640 10 client.?.objecter ms_handle_connect 0x7f3c3414ff40 2024-03-29T11:38:21.574 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.565+0000 7f3c22ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:21.574 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.565+0000 7f3c22ffd640 10 client.?.objecter ms_handle_connect 0x7f3c340a9930 2024-03-29T11:38:21.574 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.565+0000 7f3c22ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:21.574 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.565+0000 7f3c22ffd640 1 -- 172.21.15.72:0/1085887425 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2884850024 0 0) 0x7f3c34144fe0 con 0x7f3c3414ff40 2024-03-29T11:38:21.574 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.565+0000 7f3c22ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:21.574 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.565+0000 7f3c22ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:21.574 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.566+0000 7f3c22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:21.575 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.566+0000 7f3c22ffd640 10 monclient(hunting): my global_id is 4344 2024-03-29T11:38:21.575 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.566+0000 7f3c22ffd640 1 -- 172.21.15.72:0/1085887425 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c14004eb0 con 0x7f3c3414ff40 2024-03-29T11:38:21.575 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.566+0000 7f3c22ffd640 1 -- 172.21.15.72:0/1085887425 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 807271750 0 0) 0x7f3c34147e50 con 0x7f3c340a9930 2024-03-29T11:38:21.575 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.566+0000 7f3c22ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:21.575 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.566+0000 7f3c22ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:21.575 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.566+0000 7f3c22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:21.575 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.566+0000 7f3c22ffd640 10 monclient(hunting): my global_id is 4346 2024-03-29T11:38:21.575 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.566+0000 7f3c22ffd640 1 -- 172.21.15.72:0/1085887425 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c34144fe0 con 0x7f3c340a9930 2024-03-29T11:38:21.575 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.566+0000 7f3c22ffd640 1 -- 172.21.15.72:0/1085887425 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3344568572 0 0) 0x7f3c14004eb0 con 0x7f3c3414ff40 2024-03-29T11:38:21.575 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.566+0000 7f3c22ffd640 1 -- 172.21.15.72:0/1085887425 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c34147e50 con 0x7f3c3414ff40 2024-03-29T11:38:21.575 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.566+0000 7f3c22ffd640 1 -- 172.21.15.72:0/1085887425 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f3c2c001c10 con 0x7f3c3414ff40 2024-03-29T11:38:21.575 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.566+0000 7f3c22ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:21.575 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.566+0000 7f3c22ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:38:21.575 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.566+0000 7f3c22ffd640 10 monclient(hunting): dump: 2024-03-29T11:38:21.576 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:38:21.576 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:38:21.576 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:21.576 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:21.576 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:21.576 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:21.576 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:38:21.576 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:38:21.576 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:38:21.576 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:21.576 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.566+0000 7f3c22ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:21.576 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.566+0000 7f3c22ffd640 1 -- 172.21.15.72:0/1085887425 <== mon.0 v1:172.21.15.72:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3111236466 0 0) 0x7f3c34147e50 con 0x7f3c3414ff40 2024-03-29T11:38:21.576 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.566+0000 7f3c22ffd640 1 -- 172.21.15.72:0/1085887425 >> v1:172.21.15.72:6790/0 conn(0x7f3c340a9930 legacy=0x7f3c341dbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:21.577 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.566+0000 7f3c22ffd640 1 -- 172.21.15.72:0/1085887425 >> v1:172.21.15.89:6789/0 conn(0x7f3c34149dc0 legacy=0x7f3c3413a450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:38:21.581 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.568+0000 7f3c22ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:38:21.581 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.568+0000 7f3c22ffd640 1 monclient: found mon.a 2024-03-29T11:38:21.581 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.568+0000 7f3c22ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:21.581 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.568+0000 7f3c22ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:38:21.581 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.568+0000 7f3c22ffd640 1 -- 172.21.15.72:0/1085887425 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c34149900 con 0x7f3c3414ff40 2024-03-29T11:38:21.581 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.568+0000 7f3c22ffd640 10 monclient: _finish_auth 0 2024-03-29T11:38:21.581 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.568+0000 7f3c22ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:38:21.581 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.568+0000 7f3c22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:21.582 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.568+0000 7f3c38ce0640 5 monclient: authenticate success, global_id 4344 2024-03-29T11:38:21.582 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.568+0000 7f3c38ce0640 10 monclient: _renew_subs 2024-03-29T11:38:21.582 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.568+0000 7f3c38ce0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:38:21.582 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.568+0000 7f3c38ce0640 1 -- 172.21.15.72:0/1085887425 --> v1:172.21.15.72:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c3413e040 con 0x7f3c3414ff40 2024-03-29T11:38:21.582 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.568+0000 7f3c38ce0640 10 client.4344.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:21.582 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.568+0000 7f3c38ce0640 10 monclient: _renew_subs 2024-03-29T11:38:21.582 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.568+0000 7f3c38ce0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:38:21.582 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.568+0000 7f3c38ce0640 1 -- 172.21.15.72:0/1085887425 --> v1:172.21.15.72:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c341dc4d0 con 0x7f3c3414ff40 2024-03-29T11:38:21.582 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.568+0000 7f3c38ce0640 1 librados: init done 2024-03-29T11:38:21.582 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.568+0000 7f3c22ffd640 1 -- 172.21.15.72:0/1085887425 <== mon.0 v1:172.21.15.72:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c2c004b60 con 0x7f3c3414ff40 2024-03-29T11:38:21.582 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.568+0000 7f3c22ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:21.582 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.568+0000 7f3c30d99640 4 set_mon_vals no callback set 2024-03-29T11:38:21.582 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.569+0000 7f3c22ffd640 1 -- 172.21.15.72:0/1085887425 <== mon.0 v1:172.21.15.72:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f3c2c001d70 con 0x7f3c3414ff40 2024-03-29T11:38:21.583 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.569+0000 7f3c22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:21.583 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.569+0000 7f3c22ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:21.583 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.569+0000 7f3c22ffd640 10 monclient: dump: 2024-03-29T11:38:21.583 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:38:21.583 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:38:21.583 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:21.583 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:21.583 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:21.583 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:21.583 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:38:21.583 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:38:21.583 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:38:21.583 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:21.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.570+0000 7f3c38ce0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:21.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.571+0000 7f3c38ce0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:21.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.571+0000 7f3c38ce0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:38:21.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.571+0000 7f3c38ce0640 1 -- 172.21.15.72:0/1085887425 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c18005080 con 0x7f3c3414ff40 2024-03-29T11:38:21.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.571+0000 7f3c22ffd640 1 -- 172.21.15.72:0/1085887425 <== mon.0 v1:172.21.15.72:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f3c2c005f20 con 0x7f3c3414ff40 2024-03-29T11:38:21.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.571+0000 7f3c22ffd640 1 -- 172.21.15.72:0/1085887425 <== mon.0 v1:172.21.15.72:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f3c2c084d30 con 0x7f3c3414ff40 2024-03-29T11:38:21.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.571+0000 7f3c22ffd640 10 client.4344.objecter ms_dispatch 0x7f3c34000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:21.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.571+0000 7f3c22ffd640 3 client.4344.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:21.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.571+0000 7f3c22ffd640 3 client.4344.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:21.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.571+0000 7f3c22ffd640 20 client.4344.objecter dump_active .. 0 homeless 2024-03-29T11:38:21.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.572+0000 7f3c3259c640 10 monclient: get_auth_request con 0x7f3c1406ddd0 auth_method 0 2024-03-29T11:38:21.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.572+0000 7f3c22ffd640 10 client.4344.objecter ms_handle_connect 0x7f3c1406ddd0 2024-03-29T11:38:21.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.579+0000 7f3c22ffd640 1 -- 172.21.15.72:0/1085887425 <== mon.0 v1:172.21.15.72:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3c2c050610 con 0x7f3c3414ff40 2024-03-29T11:38:21.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.579+0000 7f3c22ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:21.585 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.579+0000 7f3c22ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:21.890 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.890+0000 7f3c38ce0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:21.891 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.890+0000 7f3c38ce0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:21.891 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.890+0000 7f3c38ce0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:38:21.891 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.890+0000 7f3c38ce0640 1 -- 172.21.15.72:0/1085887425 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c18005330 con 0x7f3c3414ff40 2024-03-29T11:38:21.891 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.891+0000 7f3c22ffd640 1 -- 172.21.15.72:0/1085887425 <== mon.0 v1:172.21.15.72:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f3c2c052540 con 0x7f3c3414ff40 2024-03-29T11:38:21.891 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.891+0000 7f3c22ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:21.891 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.891+0000 7f3c22ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:21.892 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:38:21.892 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:21.894 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.894+0000 7f3c38ce0640 10 librados: watch_flush enter 2024-03-29T11:38:21.895 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.894+0000 7f3c38ce0640 10 librados: watch_flush exit 2024-03-29T11:38:21.895 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.894+0000 7f3c38ce0640 20 client.4344.objecter shutdown clearing up homeless session... 2024-03-29T11:38:21.895 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.894+0000 7f3c38ce0640 10 client.4344.objecter successfully canceled tick 2024-03-29T11:38:21.895 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.894+0000 7f3c38ce0640 1 -- 172.21.15.72:0/1085887425 >> v1:172.21.15.72:6816/1578866102 conn(0x7f3c1406ddd0 legacy=0x7f3c14070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:21.895 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.894+0000 7f3c38ce0640 10 monclient: shutdown 2024-03-29T11:38:21.895 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.894+0000 7f3c38ce0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:21.896 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.894+0000 7f3c38ce0640 1 -- 172.21.15.72:0/1085887425 >> v1:172.21.15.72:6789/0 conn(0x7f3c3414ff40 legacy=0x7f3c341e6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:21.896 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.894+0000 7f3c38ce0640 1 -- 172.21.15.72:0/1085887425 shutdown_connections 2024-03-29T11:38:21.896 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.894+0000 7f3c38ce0640 1 -- 172.21.15.72:0/1085887425 >> 172.21.15.72:0/1085887425 conn(0x7f3c340b35e0 msgr2=0x7f3c340b1e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:21.896 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.895+0000 7f3c38ce0640 1 -- 172.21.15.72:0/1085887425 shutdown_connections 2024-03-29T11:38:21.896 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.895+0000 7f3c38ce0640 1 -- 172.21.15.72:0/1085887425 wait complete. 2024-03-29T11:38:21.896 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:21.895+0000 7f3c38ce0640 1 librados: shutdown 2024-03-29T11:38:21.909 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:28.716 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:28.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.859+0000 7fcc12304640 10 monclient: get_monmap_and_config 2024-03-29T11:38:28.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.859+0000 7fcc12304640 10 monclient: build_initial_monmap 2024-03-29T11:38:28.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.860+0000 7fcc12304640 10 monclient: monmap: 2024-03-29T11:38:28.860 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:38:28.860 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:38:28.861 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:38:28.860965+0000 2024-03-29T11:38:28.861 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:38:28.860965+0000 2024-03-29T11:38:28.861 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:28.861 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:28.861 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:38:28.861 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:38:28.861 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:38:28.861 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:28.861 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.861+0000 7fcc12304640 1 Processor -- start 2024-03-29T11:38:28.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.861+0000 7fcc12304640 1 -- start start 2024-03-29T11:38:28.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.861+0000 7fcc12304640 10 monclient: init 2024-03-29T11:38:28.863 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.862+0000 7fcc12304640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:28.863 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.862+0000 7fcc12304640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:38:28.863 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.862+0000 7fcc12304640 10 monclient(hunting): picked mon.noname-b con 0x7fcc0c0b1f60 addr v1:172.21.15.72:6790/0 2024-03-29T11:38:28.863 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.862+0000 7fcc12304640 10 monclient(hunting): picked mon.noname-c con 0x7fcc0c071eb0 addr v1:172.21.15.89:6789/0 2024-03-29T11:38:28.863 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.862+0000 7fcc12304640 10 monclient(hunting): picked mon.noname-a con 0x7fcc0c143d10 addr v1:172.21.15.72:6789/0 2024-03-29T11:38:28.863 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.862+0000 7fcc12304640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc0c06ef80 con 0x7fcc0c143d10 2024-03-29T11:38:28.863 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.862+0000 7fcc12304640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc0c06f260 con 0x7fcc0c071eb0 2024-03-29T11:38:28.863 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.862+0000 7fcc12304640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc0c0b40c0 con 0x7fcc0c0b1f60 2024-03-29T11:38:28.863 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.862+0000 7fcc12304640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:28.863 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.862+0000 7fcc12304640 10 monclient(hunting): authenticate will time out at 1068.150181s 2024-03-29T11:38:28.863 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.862+0000 7fcc0b7fe640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fcc0c071eb0 0x7fcc0c146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:44860/0 (socket says 172.21.15.72:44860) 2024-03-29T11:38:28.863 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.862+0000 7fcc0b7fe640 1 -- 172.21.15.72:0/1451029594 learned_addr learned my addr 172.21.15.72:0/1451029594 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:38:28.864 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.862+0000 7fcc0affd640 1 -- 172.21.15.72:0/1451029594 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1476648724 0 0) 0x7fcc0c06f260 con 0x7fcc0c071eb0 2024-03-29T11:38:28.864 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.862+0000 7fcc0affd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:28.864 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.862+0000 7fcc0affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:28.864 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.862+0000 7fcc0affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:28.864 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.862+0000 7fcc0affd640 10 monclient(hunting): my global_id is 4336 2024-03-29T11:38:28.864 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.862+0000 7fcc0affd640 1 -- 172.21.15.72:0/1451029594 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbf4003520 con 0x7fcc0c071eb0 2024-03-29T11:38:28.866 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.863+0000 7fcc0affd640 1 -- 172.21.15.72:0/1451029594 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 411033039 0 0) 0x7fcc0c06ef80 con 0x7fcc0c143d10 2024-03-29T11:38:28.867 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.863+0000 7fcc0affd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:28.867 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.863+0000 7fcc0affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:28.867 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.863+0000 7fcc0affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:28.867 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.863+0000 7fcc0affd640 10 monclient(hunting): my global_id is 4347 2024-03-29T11:38:28.867 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.863+0000 7fcc0affd640 1 -- 172.21.15.72:0/1451029594 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc0c06f260 con 0x7fcc0c143d10 2024-03-29T11:38:28.867 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.863+0000 7fcc0affd640 1 -- 172.21.15.72:0/1451029594 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2254817927 0 0) 0x7fcc0c0b40c0 con 0x7fcc0c0b1f60 2024-03-29T11:38:28.867 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.863+0000 7fcc0affd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:28.867 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.863+0000 7fcc0affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:28.867 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.863+0000 7fcc0affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:28.867 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.863+0000 7fcc0affd640 10 monclient(hunting): my global_id is 4349 2024-03-29T11:38:28.867 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.863+0000 7fcc0affd640 1 -- 172.21.15.72:0/1451029594 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc0c06ef80 con 0x7fcc0c0b1f60 2024-03-29T11:38:28.867 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.863+0000 7fcc0affd640 1 -- 172.21.15.72:0/1451029594 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3221880017 0 0) 0x7fcc0c06ef80 con 0x7fcc0c0b1f60 2024-03-29T11:38:28.868 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.863+0000 7fcc0affd640 1 -- 172.21.15.72:0/1451029594 >> v1:172.21.15.89:6789/0 conn(0x7fcc0c071eb0 legacy=0x7fcc0c146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:28.868 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.863+0000 7fcc0affd640 1 -- 172.21.15.72:0/1451029594 >> v1:172.21.15.72:6789/0 conn(0x7fcc0c143d10 legacy=0x7fcc0c146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:28.868 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.863+0000 7fcc0affd640 10 monclient: _finish_hunting 0 2024-03-29T11:38:28.868 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.863+0000 7fcc0affd640 1 monclient: found mon.noname-b 2024-03-29T11:38:28.868 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.863+0000 7fcc0affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:28.868 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.863+0000 7fcc0affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.72:6790/0 2024-03-29T11:38:28.868 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.863+0000 7fcc0affd640 1 -- 172.21.15.72:0/1451029594 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc0c155d70 con 0x7fcc0c0b1f60 2024-03-29T11:38:28.868 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.863+0000 7fcc0affd640 10 monclient: _finish_auth 0 2024-03-29T11:38:28.868 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.863+0000 7fcc0affd640 10 monclient: _check_auth_tickets 2024-03-29T11:38:28.868 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.863+0000 7fcc0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:28.868 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.863+0000 7fcc0affd640 1 -- 172.21.15.72:0/1451029594 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fcc040017e0 con 0x7fcc0c0b1f60 2024-03-29T11:38:28.868 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.863+0000 7fcc0affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:28.868 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.863+0000 7fcc0affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:28.868 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.863+0000 7fcc0affd640 10 monclient: dump: 2024-03-29T11:38:28.869 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:38:28.869 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:38:28.869 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:28.869 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:28.869 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:28.869 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:28.869 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:38:28.869 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:38:28.869 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:38:28.869 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:28.869 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.863+0000 7fcc12304640 5 monclient: authenticate success, global_id 4349 2024-03-29T11:38:28.869 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.863+0000 7fcc12304640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:28.869 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.863+0000 7fcc0affd640 1 -- 172.21.15.72:0/1451029594 <== mon.2 v1:172.21.15.72:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc04001940 con 0x7fcc0c0b1f60 2024-03-29T11:38:28.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.863+0000 7fcc0affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:28.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.863+0000 7fcc12304640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:28.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.863+0000 7fcc0affd640 1 -- 172.21.15.72:0/1451029594 <== mon.2 v1:172.21.15.72:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fcc04001d90 con 0x7fcc0c0b1f60 2024-03-29T11:38:28.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.863+0000 7fcc12304640 4 set_mon_vals no callback set 2024-03-29T11:38:28.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.863+0000 7fcc0affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:28.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.863+0000 7fcc0affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:28.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.863+0000 7fcc0affd640 10 monclient: dump: 2024-03-29T11:38:28.870 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:38:28.870 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:38:28.870 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:28.870 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:28.870 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:28.871 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:28.871 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:38:28.871 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:38:28.871 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:38:28.871 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:28.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.864+0000 7fcc12304640 10 monclient: shutdown 2024-03-29T11:38:28.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.864+0000 7fcc12304640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:28.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.864+0000 7fcc12304640 1 -- 172.21.15.72:0/1451029594 >> v1:172.21.15.72:6790/0 conn(0x7fcc0c0b1f60 legacy=0x7fcc0c143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:28.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.864+0000 7fcc12304640 1 -- 172.21.15.72:0/1451029594 shutdown_connections 2024-03-29T11:38:28.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.864+0000 7fcc12304640 1 -- 172.21.15.72:0/1451029594 >> 172.21.15.72:0/1451029594 conn(0x7fcc0c066a00 msgr2=0x7fcc0c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:28.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.864+0000 7fcc12304640 1 -- 172.21.15.72:0/1451029594 shutdown_connections 2024-03-29T11:38:28.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.865+0000 7fcc12304640 1 -- 172.21.15.72:0/1451029594 wait complete. 2024-03-29T11:38:28.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.865+0000 7fcc12304640 10 monclient: build_initial_monmap 2024-03-29T11:38:28.872 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.865+0000 7fcc12304640 10 monclient: monmap: 2024-03-29T11:38:28.872 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:38:28.872 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:28.872 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:38:28.872 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:38:28.872 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:28.872 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:28.872 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:38:28.872 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:38:28.872 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:38:28.872 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:28.872 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.866+0000 7fcc12304640 1 librados: starting msgr at 2024-03-29T11:38:28.872 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.866+0000 7fcc12304640 1 librados: starting objecter 2024-03-29T11:38:28.872 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.866+0000 7fcc12304640 1 Processor -- start 2024-03-29T11:38:28.878 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.866+0000 7fcc12304640 1 -- start start 2024-03-29T11:38:28.878 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.866+0000 7fcc12304640 1 librados: setting wanted keys 2024-03-29T11:38:28.878 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.866+0000 7fcc12304640 1 librados: calling monclient init 2024-03-29T11:38:28.878 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.866+0000 7fcc12304640 10 monclient: init 2024-03-29T11:38:28.878 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.867+0000 7fcc12304640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:28.879 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.867+0000 7fcc12304640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:38:28.879 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.867+0000 7fcc12304640 10 monclient(hunting): picked mon.noname-a con 0x7fcc0c0b1f60 addr v1:172.21.15.72:6789/0 2024-03-29T11:38:28.879 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.868+0000 7fcc12304640 10 monclient(hunting): picked mon.noname-b con 0x7fcc0c143d10 addr v1:172.21.15.89:6789/0 2024-03-29T11:38:28.879 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.868+0000 7fcc0bfff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fcc0c0b1f60 0x7fcc0c06f6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:58608/0 (socket says 172.21.15.72:58608) 2024-03-29T11:38:28.879 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.868+0000 7fcc12304640 10 monclient(hunting): picked mon.noname-c con 0x7fcc0c136f90 addr v1:172.21.15.72:6790/0 2024-03-29T11:38:28.879 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.868+0000 7fcc0bfff640 1 -- 172.21.15.72:0/3483395198 learned_addr learned my addr 172.21.15.72:0/3483395198 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:38:28.879 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.868+0000 7fcc12304640 1 -- 172.21.15.72:0/3483395198 --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc0c153500 con 0x7fcc0c0b1f60 2024-03-29T11:38:28.879 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.868+0000 7fcc12304640 1 -- 172.21.15.72:0/3483395198 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc0c154740 con 0x7fcc0c143d10 2024-03-29T11:38:28.879 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.868+0000 7fcc12304640 1 -- 172.21.15.72:0/3483395198 --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc0c154a20 con 0x7fcc0c136f90 2024-03-29T11:38:28.879 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.868+0000 7fcc12304640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:28.879 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.868+0000 7fcc12304640 10 monclient(hunting): authenticate will time out at 1068.156732s 2024-03-29T11:38:28.879 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.868+0000 7fcc08ff9640 10 client.?.objecter ms_handle_connect 0x7fcc0c136f90 2024-03-29T11:38:28.880 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.868+0000 7fcc08ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:28.880 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.868+0000 7fcc08ff9640 10 client.?.objecter ms_handle_connect 0x7fcc0c0b1f60 2024-03-29T11:38:28.880 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.868+0000 7fcc08ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:28.880 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.869+0000 7fcc08ff9640 1 -- 172.21.15.72:0/3483395198 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3915650954 0 0) 0x7fcc0c153500 con 0x7fcc0c0b1f60 2024-03-29T11:38:28.880 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.869+0000 7fcc08ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:38:28.880 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.869+0000 7fcc08ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:28.880 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.869+0000 7fcc08ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:28.880 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.869+0000 7fcc08ff9640 10 monclient(hunting): my global_id is 4350 2024-03-29T11:38:28.880 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.869+0000 7fcc08ff9640 1 -- 172.21.15.72:0/3483395198 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbf8004eb0 con 0x7fcc0c0b1f60 2024-03-29T11:38:28.880 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.869+0000 7fcc08ff9640 1 -- 172.21.15.72:0/3483395198 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3691911837 0 0) 0x7fcc0c154a20 con 0x7fcc0c136f90 2024-03-29T11:38:28.880 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.869+0000 7fcc08ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:38:28.881 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.869+0000 7fcc08ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:28.881 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.869+0000 7fcc08ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:28.881 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.869+0000 7fcc08ff9640 10 monclient(hunting): my global_id is 4352 2024-03-29T11:38:28.881 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.869+0000 7fcc08ff9640 1 -- 172.21.15.72:0/3483395198 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc0c153500 con 0x7fcc0c136f90 2024-03-29T11:38:28.881 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.869+0000 7fcc08ff9640 1 -- 172.21.15.72:0/3483395198 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 150831543 0 0) 0x7fcbf8004eb0 con 0x7fcc0c0b1f60 2024-03-29T11:38:28.881 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.869+0000 7fcc08ff9640 1 -- 172.21.15.72:0/3483395198 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc0c154a20 con 0x7fcc0c0b1f60 2024-03-29T11:38:28.881 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.869+0000 7fcc08ff9640 1 -- 172.21.15.72:0/3483395198 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fcc04001c90 con 0x7fcc0c0b1f60 2024-03-29T11:38:28.881 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.869+0000 7fcc08ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:28.881 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.869+0000 7fcc08ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:38:28.881 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.869+0000 7fcc08ff9640 10 monclient(hunting): dump: 2024-03-29T11:38:28.881 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:38:28.881 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:38:28.881 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:28.881 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:28.882 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:28.882 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:28.882 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:38:28.882 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:38:28.882 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:38:28.882 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:28.882 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.869+0000 7fcc08ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:28.882 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.869+0000 7fcc08ff9640 1 -- 172.21.15.72:0/3483395198 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 376972713 0 0) 0x7fcc0c153500 con 0x7fcc0c136f90 2024-03-29T11:38:28.882 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.870+0000 7fcc08ff9640 1 -- 172.21.15.72:0/3483395198 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcbf8004eb0 con 0x7fcc0c136f90 2024-03-29T11:38:28.882 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.870+0000 7fcc08ff9640 1 -- 172.21.15.72:0/3483395198 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fcc00001850 con 0x7fcc0c136f90 2024-03-29T11:38:28.882 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.870+0000 7fcc08ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:28.882 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.870+0000 7fcc08ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:28.883 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.870+0000 7fcc08ff9640 10 monclient(hunting): dump: 2024-03-29T11:38:28.883 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:38:28.883 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:38:28.883 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:28.883 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:28.883 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:28.883 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:28.883 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:38:28.883 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:38:28.883 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:38:28.883 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:28.883 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.870+0000 7fcc08ff9640 10 client.?.objecter ms_handle_connect 0x7fcc0c143d10 2024-03-29T11:38:28.883 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.870+0000 7fcc08ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:28.884 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.870+0000 7fcc08ff9640 1 -- 172.21.15.72:0/3483395198 <== mon.0 v1:172.21.15.72:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1632994289 0 0) 0x7fcc0c154a20 con 0x7fcc0c0b1f60 2024-03-29T11:38:28.884 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.870+0000 7fcc08ff9640 1 -- 172.21.15.72:0/3483395198 >> v1:172.21.15.72:6790/0 conn(0x7fcc0c136f90 legacy=0x7fcc0c1e60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:28.884 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.870+0000 7fcc08ff9640 1 -- 172.21.15.72:0/3483395198 >> v1:172.21.15.89:6789/0 conn(0x7fcc0c143d10 legacy=0x7fcc0c06fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:28.884 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.870+0000 7fcc08ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:38:28.884 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.870+0000 7fcc08ff9640 1 monclient: found mon.a 2024-03-29T11:38:28.884 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.870+0000 7fcc08ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:28.884 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.870+0000 7fcc08ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:38:28.884 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.870+0000 7fcc08ff9640 1 -- 172.21.15.72:0/3483395198 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc0c154d00 con 0x7fcc0c0b1f60 2024-03-29T11:38:28.884 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.870+0000 7fcc08ff9640 10 monclient: _finish_auth 0 2024-03-29T11:38:28.884 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.870+0000 7fcc08ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:38:28.884 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.870+0000 7fcc08ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:28.884 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.870+0000 7fcc12304640 5 monclient: authenticate success, global_id 4350 2024-03-29T11:38:28.884 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.870+0000 7fcc12304640 10 monclient: _renew_subs 2024-03-29T11:38:28.885 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.870+0000 7fcc12304640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:38:28.885 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.870+0000 7fcc12304640 1 -- 172.21.15.72:0/3483395198 --> v1:172.21.15.72:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc0c155400 con 0x7fcc0c0b1f60 2024-03-29T11:38:28.885 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.870+0000 7fcc12304640 10 client.4350.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:28.885 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.870+0000 7fcc12304640 10 monclient: _renew_subs 2024-03-29T11:38:28.885 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.870+0000 7fcc12304640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:38:28.885 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.871+0000 7fcc12304640 1 -- 172.21.15.72:0/3483395198 --> v1:172.21.15.72:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc0c1e79b0 con 0x7fcc0c0b1f60 2024-03-29T11:38:28.885 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.871+0000 7fcc12304640 1 librados: init done 2024-03-29T11:38:28.885 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.871+0000 7fcc08ff9640 1 -- 172.21.15.72:0/3483395198 <== mon.0 v1:172.21.15.72:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc04003c20 con 0x7fcc0c0b1f60 2024-03-29T11:38:28.885 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.871+0000 7fcc08ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:28.885 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.871+0000 7fcc08ff9640 1 -- 172.21.15.72:0/3483395198 <== mon.0 v1:172.21.15.72:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fcc04003fb0 con 0x7fcc0c0b1f60 2024-03-29T11:38:28.885 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.871+0000 7fcc08ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:28.886 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.871+0000 7fcc08ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:28.886 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.871+0000 7fcc08ff9640 10 monclient: dump: 2024-03-29T11:38:28.886 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:38:28.886 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:38:28.886 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:28.886 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:28.886 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:28.886 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:28.886 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:38:28.886 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:38:28.886 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:38:28.886 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:28.886 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.871+0000 7fcc0a7fc640 4 set_mon_vals no callback set 2024-03-29T11:38:28.887 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.873+0000 7fcc08ff9640 1 -- 172.21.15.72:0/3483395198 <== mon.0 v1:172.21.15.72:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7fcc04004110 con 0x7fcc0c0b1f60 2024-03-29T11:38:28.887 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.873+0000 7fcc12304640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:28.887 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.873+0000 7fcc12304640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:28.887 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.873+0000 7fcc12304640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:38:28.887 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.873+0000 7fcc12304640 1 -- 172.21.15.72:0/3483395198 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc0c155700 con 0x7fcc0c0b1f60 2024-03-29T11:38:28.887 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.874+0000 7fcc08ff9640 1 -- 172.21.15.72:0/3483395198 <== mon.0 v1:172.21.15.72:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7fcc04084e40 con 0x7fcc0c0b1f60 2024-03-29T11:38:28.887 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.874+0000 7fcc08ff9640 10 client.4350.objecter ms_dispatch 0x7fcc0c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:28.887 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.874+0000 7fcc08ff9640 3 client.4350.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:28.887 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.874+0000 7fcc08ff9640 3 client.4350.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:28.887 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.874+0000 7fcc0b7fe640 10 monclient: get_auth_request con 0x7fcbf806df80 auth_method 0 2024-03-29T11:38:28.887 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.874+0000 7fcc08ff9640 20 client.4350.objecter dump_active .. 0 homeless 2024-03-29T11:38:28.887 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.875+0000 7fcc08ff9640 10 client.4350.objecter ms_handle_connect 0x7fcbf806df80 2024-03-29T11:38:28.888 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.882+0000 7fcc08ff9640 1 -- 172.21.15.72:0/3483395198 <== mon.0 v1:172.21.15.72:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcc04050720 con 0x7fcc0c0b1f60 2024-03-29T11:38:28.888 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.882+0000 7fcc08ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:28.888 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:28.882+0000 7fcc08ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:29.259 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:29.258+0000 7fcc12304640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:29.259 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:29.258+0000 7fcc12304640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:29.259 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:29.258+0000 7fcc12304640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:38:29.259 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:29.258+0000 7fcc12304640 1 -- 172.21.15.72:0/3483395198 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc0c075f10 con 0x7fcc0c0b1f60 2024-03-29T11:38:29.260 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:29.260+0000 7fcc08ff9640 1 -- 172.21.15.72:0/3483395198 <== mon.0 v1:172.21.15.72:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7fcc04052650 con 0x7fcc0c0b1f60 2024-03-29T11:38:29.260 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:29.260+0000 7fcc08ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:29.261 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:29.260+0000 7fcc08ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:29.261 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:38:29.261 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:29.264 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:29.263+0000 7fcc12304640 10 librados: watch_flush enter 2024-03-29T11:38:29.264 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:29.263+0000 7fcc12304640 10 librados: watch_flush exit 2024-03-29T11:38:29.264 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:29.263+0000 7fcc12304640 20 client.4350.objecter shutdown clearing up homeless session... 2024-03-29T11:38:29.264 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:29.263+0000 7fcc12304640 10 client.4350.objecter successfully canceled tick 2024-03-29T11:38:29.264 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:29.263+0000 7fcc12304640 1 -- 172.21.15.72:0/3483395198 >> v1:172.21.15.72:6816/1578866102 conn(0x7fcbf806df80 legacy=0x7fcbf8070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:29.264 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:29.263+0000 7fcc12304640 10 monclient: shutdown 2024-03-29T11:38:29.264 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:29.263+0000 7fcc12304640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:29.264 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:29.263+0000 7fcc12304640 1 -- 172.21.15.72:0/3483395198 >> v1:172.21.15.72:6789/0 conn(0x7fcc0c0b1f60 legacy=0x7fcc0c06f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:29.264 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:29.263+0000 7fcc12304640 1 -- 172.21.15.72:0/3483395198 shutdown_connections 2024-03-29T11:38:29.265 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:29.263+0000 7fcc12304640 1 -- 172.21.15.72:0/3483395198 >> 172.21.15.72:0/3483395198 conn(0x7fcc0c066a00 msgr2=0x7fcc0c0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:29.265 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:29.263+0000 7fcc12304640 1 -- 172.21.15.72:0/3483395198 shutdown_connections 2024-03-29T11:38:29.265 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:29.264+0000 7fcc12304640 1 -- 172.21.15.72:0/3483395198 wait complete. 2024-03-29T11:38:29.265 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:29.264+0000 7fcc12304640 1 librados: shutdown 2024-03-29T11:38:29.278 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:36.084 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:36.220 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.219+0000 7f36481a3640 10 monclient: get_monmap_and_config 2024-03-29T11:38:36.220 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.219+0000 7f36481a3640 10 monclient: build_initial_monmap 2024-03-29T11:38:36.220 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.219+0000 7f36481a3640 10 monclient: monmap: 2024-03-29T11:38:36.220 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:38:36.221 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:38:36.221 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:38:36.220416+0000 2024-03-29T11:38:36.221 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:38:36.220416+0000 2024-03-29T11:38:36.221 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:36.221 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:36.221 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:38:36.221 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:38:36.221 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:38:36.221 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:36.221 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.220+0000 7f36481a3640 1 Processor -- start 2024-03-29T11:38:36.223 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.220+0000 7f36481a3640 1 -- start start 2024-03-29T11:38:36.223 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.220+0000 7f36481a3640 10 monclient: init 2024-03-29T11:38:36.223 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.221+0000 7f36481a3640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:36.223 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.221+0000 7f36481a3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:38:36.223 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.221+0000 7f36481a3640 10 monclient(hunting): picked mon.noname-a con 0x7f36400ae250 addr v1:172.21.15.72:6789/0 2024-03-29T11:38:36.223 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.221+0000 7f36481a3640 10 monclient(hunting): picked mon.noname-b con 0x7f364013f970 addr v1:172.21.15.72:6790/0 2024-03-29T11:38:36.223 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.221+0000 7f36481a3640 10 monclient(hunting): picked mon.noname-c con 0x7f3640142800 addr v1:172.21.15.89:6789/0 2024-03-29T11:38:36.223 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.221+0000 7f36481a3640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36400afe00 con 0x7f36400ae250 2024-03-29T11:38:36.223 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.221+0000 7f36481a3640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3640142c10 con 0x7f3640142800 2024-03-29T11:38:36.223 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.221+0000 7f36481a3640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36400afb20 con 0x7f364013f970 2024-03-29T11:38:36.223 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.221+0000 7f36481a3640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:36.224 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.221+0000 7f36481a3640 10 monclient(hunting): authenticate will time out at 1075.509303s 2024-03-29T11:38:36.224 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.221+0000 7f364573e640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f364013f970 0x7f36401420f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:33724/0 (socket says 172.21.15.72:33724) 2024-03-29T11:38:36.224 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.221+0000 7f364573e640 1 -- 172.21.15.72:0/3785208552 learned_addr learned my addr 172.21.15.72:0/3785208552 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:38:36.224 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.221+0000 7f3644f3d640 1 -- 172.21.15.72:0/3785208552 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2674019345 0 0) 0x7f36400afb20 con 0x7f364013f970 2024-03-29T11:38:36.224 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.221+0000 7f3644f3d640 10 monclient(hunting): _negotiate 2024-03-29T11:38:36.224 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.221+0000 7f3644f3d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:36.224 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.221+0000 7f3644f3d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:36.224 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.221+0000 7f3644f3d640 10 monclient(hunting): my global_id is 4355 2024-03-29T11:38:36.224 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.221+0000 7f3644f3d640 1 -- 172.21.15.72:0/3785208552 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f362c003520 con 0x7f364013f970 2024-03-29T11:38:36.224 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.221+0000 7f3644f3d640 1 -- 172.21.15.72:0/3785208552 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 760766848 0 0) 0x7f3640142c10 con 0x7f3640142800 2024-03-29T11:38:36.224 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.221+0000 7f3644f3d640 10 monclient(hunting): _negotiate 2024-03-29T11:38:36.224 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.221+0000 7f3644f3d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:36.224 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.221+0000 7f3644f3d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:36.224 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.221+0000 7f3644f3d640 10 monclient(hunting): my global_id is 4342 2024-03-29T11:38:36.225 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.221+0000 7f3644f3d640 1 -- 172.21.15.72:0/3785208552 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36400afb20 con 0x7f3640142800 2024-03-29T11:38:36.225 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.222+0000 7f3644f3d640 1 -- 172.21.15.72:0/3785208552 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2006379659 0 0) 0x7f362c003520 con 0x7f364013f970 2024-03-29T11:38:36.225 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.222+0000 7f3644f3d640 1 -- 172.21.15.72:0/3785208552 >> v1:172.21.15.89:6789/0 conn(0x7f3640142800 legacy=0x7f364014da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:36.225 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.222+0000 7f3644f3d640 1 -- 172.21.15.72:0/3785208552 >> v1:172.21.15.72:6789/0 conn(0x7f36400ae250 legacy=0x7f364013f260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:38:36.228 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.223+0000 7f3644f3d640 10 monclient: _finish_hunting 0 2024-03-29T11:38:36.228 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.223+0000 7f3644f3d640 1 monclient: found mon.noname-b 2024-03-29T11:38:36.228 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.223+0000 7f3644f3d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:36.228 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.223+0000 7f3644f3d640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.72:6790/0 2024-03-29T11:38:36.228 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.223+0000 7f3644f3d640 1 -- 172.21.15.72:0/3785208552 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36400aab80 con 0x7f364013f970 2024-03-29T11:38:36.228 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.223+0000 7f3644f3d640 10 monclient: _finish_auth 0 2024-03-29T11:38:36.228 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.223+0000 7f3644f3d640 10 monclient: _check_auth_tickets 2024-03-29T11:38:36.228 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.223+0000 7f3644f3d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:36.229 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.223+0000 7f3644f3d640 1 -- 172.21.15.72:0/3785208552 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f3630001690 con 0x7f364013f970 2024-03-29T11:38:36.229 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.223+0000 7f3644f3d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:36.229 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.224+0000 7f3644f3d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:36.229 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.224+0000 7f3644f3d640 10 monclient: dump: 2024-03-29T11:38:36.229 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:38:36.229 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:38:36.229 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:36.229 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:36.229 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:36.229 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:36.229 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:38:36.229 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:38:36.229 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:38:36.229 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:36.229 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.224+0000 7f36481a3640 5 monclient: authenticate success, global_id 4355 2024-03-29T11:38:36.230 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.224+0000 7f36481a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:36.230 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.224+0000 7f3644f3d640 1 -- 172.21.15.72:0/3785208552 <== mon.2 v1:172.21.15.72:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3630001870 con 0x7f364013f970 2024-03-29T11:38:36.230 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.224+0000 7f3644f3d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:36.230 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.224+0000 7f36481a3640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:36.230 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.224+0000 7f36481a3640 4 set_mon_vals no callback set 2024-03-29T11:38:36.230 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.225+0000 7f3644f3d640 1 -- 172.21.15.72:0/3785208552 <== mon.2 v1:172.21.15.72:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f3630001c80 con 0x7f364013f970 2024-03-29T11:38:36.230 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.225+0000 7f3644f3d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:36.230 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.225+0000 7f3644f3d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:36.230 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.225+0000 7f3644f3d640 10 monclient: dump: 2024-03-29T11:38:36.230 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:38:36.230 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:38:36.230 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:36.230 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:36.230 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:36.230 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:36.231 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:38:36.231 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:38:36.231 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:38:36.231 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:36.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.225+0000 7f36481a3640 10 monclient: shutdown 2024-03-29T11:38:36.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.225+0000 7f36481a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:36.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.225+0000 7f36481a3640 1 -- 172.21.15.72:0/3785208552 >> v1:172.21.15.72:6790/0 conn(0x7f364013f970 legacy=0x7f36401420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:36.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.226+0000 7f36481a3640 1 -- 172.21.15.72:0/3785208552 shutdown_connections 2024-03-29T11:38:36.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.226+0000 7f36481a3640 1 -- 172.21.15.72:0/3785208552 >> 172.21.15.72:0/3785208552 conn(0x7f36400b35e0 msgr2=0x7f36400b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:36.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.226+0000 7f36481a3640 1 -- 172.21.15.72:0/3785208552 shutdown_connections 2024-03-29T11:38:36.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.226+0000 7f36481a3640 1 -- 172.21.15.72:0/3785208552 wait complete. 2024-03-29T11:38:36.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.227+0000 7f36481a3640 10 monclient: build_initial_monmap 2024-03-29T11:38:36.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.227+0000 7f36481a3640 10 monclient: monmap: 2024-03-29T11:38:36.231 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:38:36.231 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:36.232 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:38:36.232 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:38:36.232 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:36.232 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:36.232 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:38:36.232 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:38:36.232 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:38:36.232 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:36.232 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.228+0000 7f36481a3640 1 librados: starting msgr at 2024-03-29T11:38:36.232 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.228+0000 7f36481a3640 1 librados: starting objecter 2024-03-29T11:38:36.232 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.228+0000 7f36481a3640 1 Processor -- start 2024-03-29T11:38:36.232 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.228+0000 7f36481a3640 1 -- start start 2024-03-29T11:38:36.232 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.228+0000 7f36481a3640 1 librados: setting wanted keys 2024-03-29T11:38:36.232 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.228+0000 7f36481a3640 1 librados: calling monclient init 2024-03-29T11:38:36.232 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.228+0000 7f36481a3640 10 monclient: init 2024-03-29T11:38:36.235 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.228+0000 7f36481a3640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:36.235 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.228+0000 7f36481a3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:38:36.235 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.229+0000 7f36481a3640 10 monclient(hunting): picked mon.noname-b con 0x7f36400ae250 addr v1:172.21.15.89:6789/0 2024-03-29T11:38:36.235 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.229+0000 7f36481a3640 10 monclient(hunting): picked mon.noname-a con 0x7f364013f970 addr v1:172.21.15.72:6789/0 2024-03-29T11:38:36.235 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.229+0000 7f36481a3640 10 monclient(hunting): picked mon.noname-c con 0x7f3640142800 addr v1:172.21.15.72:6790/0 2024-03-29T11:38:36.235 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.229+0000 7f36481a3640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36401364f0 con 0x7f364013f970 2024-03-29T11:38:36.235 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.229+0000 7f36481a3640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3640144290 con 0x7f36400ae250 2024-03-29T11:38:36.235 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.229+0000 7f36481a3640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f364013c0a0 con 0x7f3640142800 2024-03-29T11:38:36.235 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.229+0000 7f36481a3640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:36.235 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.229+0000 7f36481a3640 10 monclient(hunting): authenticate will time out at 1075.517156s 2024-03-29T11:38:36.235 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.229+0000 7f3646740640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f3640142800 0x7f36401db740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:34268/0 (socket says 172.21.15.72:34268) 2024-03-29T11:38:36.236 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.229+0000 7f3646740640 1 -- 172.21.15.72:0/901234538 learned_addr learned my addr 172.21.15.72:0/901234538 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:38:36.236 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.229+0000 7f36367fc640 10 client.?.objecter ms_handle_connect 0x7f3640142800 2024-03-29T11:38:36.236 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.229+0000 7f36367fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:36.236 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.229+0000 7f36367fc640 10 client.?.objecter ms_handle_connect 0x7f36400ae250 2024-03-29T11:38:36.236 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.229+0000 7f36367fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:36.236 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.229+0000 7f36367fc640 1 -- 172.21.15.72:0/901234538 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2277851817 0 0) 0x7f364013c0a0 con 0x7f3640142800 2024-03-29T11:38:36.236 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.229+0000 7f36367fc640 10 monclient(hunting): _negotiate 2024-03-29T11:38:36.236 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.229+0000 7f36367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:36.236 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.229+0000 7f36367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:36.236 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.229+0000 7f36367fc640 10 monclient(hunting): my global_id is 4358 2024-03-29T11:38:36.236 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.229+0000 7f36367fc640 1 -- 172.21.15.72:0/901234538 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3624004eb0 con 0x7f3640142800 2024-03-29T11:38:36.236 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.229+0000 7f36367fc640 1 -- 172.21.15.72:0/901234538 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2941786409 0 0) 0x7f3640144290 con 0x7f36400ae250 2024-03-29T11:38:36.236 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.229+0000 7f36367fc640 10 monclient(hunting): _negotiate 2024-03-29T11:38:36.236 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.229+0000 7f36367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:36.237 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.229+0000 7f36367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:36.237 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.229+0000 7f36367fc640 10 monclient(hunting): my global_id is 4345 2024-03-29T11:38:36.237 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.229+0000 7f36367fc640 1 -- 172.21.15.72:0/901234538 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f364013c0a0 con 0x7f36400ae250 2024-03-29T11:38:36.237 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.229+0000 7f36367fc640 1 -- 172.21.15.72:0/901234538 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1246218063 0 0) 0x7f364013c0a0 con 0x7f36400ae250 2024-03-29T11:38:36.237 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.229+0000 7f36367fc640 1 -- 172.21.15.72:0/901234538 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3640144290 con 0x7f36400ae250 2024-03-29T11:38:36.237 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.229+0000 7f36367fc640 1 -- 172.21.15.72:0/901234538 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f3638001bb0 con 0x7f36400ae250 2024-03-29T11:38:36.237 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.229+0000 7f36367fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:36.237 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.229+0000 7f36367fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:36.237 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.229+0000 7f36367fc640 10 monclient(hunting): dump: 2024-03-29T11:38:36.237 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:38:36.237 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:38:36.237 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:36.237 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:36.237 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:36.237 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:36.237 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:38:36.237 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:38:36.238 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:38:36.238 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:36.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.229+0000 7f36367fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:36.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.230+0000 7f36367fc640 1 -- 172.21.15.72:0/901234538 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2115128820 0 0) 0x7f3624004eb0 con 0x7f3640142800 2024-03-29T11:38:36.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.230+0000 7f36367fc640 1 -- 172.21.15.72:0/901234538 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f364013c0a0 con 0x7f3640142800 2024-03-29T11:38:36.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.230+0000 7f36367fc640 1 -- 172.21.15.72:0/901234538 <== mon.1 v1:172.21.15.89:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1129688413 0 0) 0x7f3640144290 con 0x7f36400ae250 2024-03-29T11:38:36.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.230+0000 7f36367fc640 1 -- 172.21.15.72:0/901234538 >> v1:172.21.15.72:6790/0 conn(0x7f3640142800 legacy=0x7f36401db740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:36.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.230+0000 7f36367fc640 1 -- 172.21.15.72:0/901234538 >> v1:172.21.15.72:6789/0 conn(0x7f364013f970 legacy=0x7f3640145ea0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:38:36.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.232+0000 7f36367fc640 10 monclient: _finish_hunting 0 2024-03-29T11:38:36.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.232+0000 7f36367fc640 1 monclient: found mon.b 2024-03-29T11:38:36.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.232+0000 7f36367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:36.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.232+0000 7f36367fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:38:36.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.232+0000 7f36367fc640 1 -- 172.21.15.72:0/901234538 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f364013c5c0 con 0x7f36400ae250 2024-03-29T11:38:36.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.232+0000 7f36367fc640 10 monclient: _finish_auth 0 2024-03-29T11:38:36.239 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.232+0000 7f36367fc640 10 monclient: _check_auth_tickets 2024-03-29T11:38:36.239 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.232+0000 7f36367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:36.239 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.232+0000 7f36481a3640 5 monclient: authenticate success, global_id 4345 2024-03-29T11:38:36.239 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.232+0000 7f36481a3640 10 monclient: _renew_subs 2024-03-29T11:38:36.239 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.232+0000 7f36481a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:38:36.239 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.232+0000 7f36481a3640 1 -- 172.21.15.72:0/901234538 --> v1:172.21.15.89:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f364013b160 con 0x7f36400ae250 2024-03-29T11:38:36.239 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.232+0000 7f36481a3640 10 client.4345.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:36.239 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.232+0000 7f36481a3640 10 monclient: _renew_subs 2024-03-29T11:38:36.239 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.232+0000 7f36481a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:38:36.239 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.232+0000 7f36481a3640 1 -- 172.21.15.72:0/901234538 --> v1:172.21.15.89:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f36401dc030 con 0x7f36400ae250 2024-03-29T11:38:36.239 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.232+0000 7f36481a3640 1 librados: init done 2024-03-29T11:38:36.239 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.233+0000 7f36481a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:36.239 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.233+0000 7f36481a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:36.239 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.233+0000 7f36481a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:38:36.239 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.233+0000 7f36481a3640 1 -- 172.21.15.72:0/901234538 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f364013c290 con 0x7f36400ae250 2024-03-29T11:38:36.240 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.234+0000 7f36367fc640 1 -- 172.21.15.72:0/901234538 <== mon.1 v1:172.21.15.89:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36380020c0 con 0x7f36400ae250 2024-03-29T11:38:36.240 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.234+0000 7f36367fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:36.240 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.234+0000 7f36367fc640 1 -- 172.21.15.72:0/901234538 <== mon.1 v1:172.21.15.89:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f3638003b10 con 0x7f36400ae250 2024-03-29T11:38:36.240 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.234+0000 7f36367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:36.240 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.234+0000 7f36367fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:36.240 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.234+0000 7f36367fc640 10 monclient: dump: 2024-03-29T11:38:36.240 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:38:36.240 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:38:36.240 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:36.240 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:36.240 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:36.240 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:36.240 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:38:36.240 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:38:36.240 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:38:36.240 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:36.241 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.234+0000 7f36377fe640 4 set_mon_vals no callback set 2024-03-29T11:38:36.241 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.236+0000 7f36367fc640 1 -- 172.21.15.72:0/901234538 <== mon.1 v1:172.21.15.89:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f3638019b60 con 0x7f36400ae250 2024-03-29T11:38:36.241 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.237+0000 7f36367fc640 1 -- 172.21.15.72:0/901234538 <== mon.1 v1:172.21.15.89:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f3638083af0 con 0x7f36400ae250 2024-03-29T11:38:36.241 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.237+0000 7f36367fc640 10 client.4345.objecter ms_dispatch 0x7f3640000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:36.241 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.237+0000 7f36367fc640 3 client.4345.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:36.241 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.237+0000 7f36367fc640 3 client.4345.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:36.241 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.237+0000 7f36367fc640 20 client.4345.objecter dump_active .. 0 homeless 2024-03-29T11:38:36.242 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.237+0000 7f364573e640 10 monclient: get_auth_request con 0x7f362406dc90 auth_method 0 2024-03-29T11:38:36.242 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.238+0000 7f36367fc640 10 client.4345.objecter ms_handle_connect 0x7f362406dc90 2024-03-29T11:38:36.243 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.243+0000 7f36367fc640 1 -- 172.21.15.72:0/901234538 <== mon.1 v1:172.21.15.89:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f363804f3d0 con 0x7f36400ae250 2024-03-29T11:38:36.243 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.243+0000 7f36367fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:36.243 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.243+0000 7f36367fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:36.551 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.550+0000 7f36481a3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:36.551 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.550+0000 7f36481a3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:36.551 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.550+0000 7f36481a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:38:36.551 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.550+0000 7f36481a3640 1 -- 172.21.15.72:0/901234538 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3640075f10 con 0x7f36400ae250 2024-03-29T11:38:36.554 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.553+0000 7f36367fc640 1 -- 172.21.15.72:0/901234538 <== mon.1 v1:172.21.15.89:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f3638051300 con 0x7f36400ae250 2024-03-29T11:38:36.554 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.553+0000 7f36367fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:36.554 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.553+0000 7f36367fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:36.554 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:38:36.554 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:36.557 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.556+0000 7f36481a3640 10 librados: watch_flush enter 2024-03-29T11:38:36.558 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.556+0000 7f36481a3640 10 librados: watch_flush exit 2024-03-29T11:38:36.558 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.556+0000 7f36481a3640 20 client.4345.objecter shutdown clearing up homeless session... 2024-03-29T11:38:36.558 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.556+0000 7f36481a3640 10 client.4345.objecter successfully canceled tick 2024-03-29T11:38:36.558 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.556+0000 7f36481a3640 1 -- 172.21.15.72:0/901234538 >> v1:172.21.15.72:6816/1578866102 conn(0x7f362406dc90 legacy=0x7f3624070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:36.558 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.556+0000 7f36481a3640 10 monclient: shutdown 2024-03-29T11:38:36.558 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.556+0000 7f36481a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:36.558 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.556+0000 7f36481a3640 1 -- 172.21.15.72:0/901234538 >> v1:172.21.15.89:6789/0 conn(0x7f36400ae250 legacy=0x7f36400ac790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:36.558 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.557+0000 7f36481a3640 1 -- 172.21.15.72:0/901234538 shutdown_connections 2024-03-29T11:38:36.558 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.557+0000 7f36481a3640 1 -- 172.21.15.72:0/901234538 >> 172.21.15.72:0/901234538 conn(0x7f36400b35e0 msgr2=0x7f36400aa030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:36.558 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.557+0000 7f36481a3640 1 -- 172.21.15.72:0/901234538 shutdown_connections 2024-03-29T11:38:36.558 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.557+0000 7f36481a3640 1 -- 172.21.15.72:0/901234538 wait complete. 2024-03-29T11:38:36.558 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:36.557+0000 7f36481a3640 1 librados: shutdown 2024-03-29T11:38:36.571 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:43.377 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:43.520 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.519+0000 7f24b9794640 10 monclient: get_monmap_and_config 2024-03-29T11:38:43.520 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.519+0000 7f24b9794640 10 monclient: build_initial_monmap 2024-03-29T11:38:43.520 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.520+0000 7f24b9794640 10 monclient: monmap: 2024-03-29T11:38:43.520 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:38:43.521 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:38:43.521 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:38:43.520974+0000 2024-03-29T11:38:43.521 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:38:43.520974+0000 2024-03-29T11:38:43.521 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:43.521 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:43.521 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:38:43.521 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:38:43.521 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:38:43.521 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:43.521 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.521+0000 7f24b9794640 1 Processor -- start 2024-03-29T11:38:43.521 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.521+0000 7f24b9794640 1 -- start start 2024-03-29T11:38:43.522 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.521+0000 7f24b9794640 10 monclient: init 2024-03-29T11:38:43.523 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.522+0000 7f24b9794640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:43.523 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.522+0000 7f24b9794640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:38:43.523 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.522+0000 7f24b9794640 10 monclient(hunting): picked mon.noname-a con 0x7f24b406fb00 addr v1:172.21.15.72:6789/0 2024-03-29T11:38:43.523 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.522+0000 7f24b9794640 10 monclient(hunting): picked mon.noname-b con 0x7f24b40b0420 addr v1:172.21.15.72:6790/0 2024-03-29T11:38:43.523 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.522+0000 7f24b9794640 10 monclient(hunting): picked mon.noname-c con 0x7f24b40af990 addr v1:172.21.15.89:6789/0 2024-03-29T11:38:43.524 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.522+0000 7f24b9794640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24b4140c10 con 0x7f24b406fb00 2024-03-29T11:38:43.524 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.522+0000 7f24b9794640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24b4071fa0 con 0x7f24b40af990 2024-03-29T11:38:43.524 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.522+0000 7f24b9794640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24b4072280 con 0x7f24b40b0420 2024-03-29T11:38:43.524 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.522+0000 7f24b9794640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:43.524 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.522+0000 7f24b9794640 10 monclient(hunting): authenticate will time out at 1082.810472s 2024-03-29T11:38:43.524 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.524+0000 7f24b37fe640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f24b40b0420 0x7f24b4140500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:36616/0 (socket says 172.21.15.72:36616) 2024-03-29T11:38:43.524 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.524+0000 7f24b37fe640 1 -- 172.21.15.72:0/3719321558 learned_addr learned my addr 172.21.15.72:0/3719321558 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:38:43.531 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.528+0000 7f24b2ffd640 1 -- 172.21.15.72:0/3719321558 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4122186318 0 0) 0x7f24b4072280 con 0x7f24b40b0420 2024-03-29T11:38:43.531 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.528+0000 7f24b2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:43.531 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.528+0000 7f24b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:43.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.528+0000 7f24b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:43.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.528+0000 7f24b2ffd640 10 monclient(hunting): my global_id is 4361 2024-03-29T11:38:43.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.528+0000 7f24b2ffd640 1 -- 172.21.15.72:0/3719321558 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f249c003520 con 0x7f24b40b0420 2024-03-29T11:38:43.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.529+0000 7f24b2ffd640 1 -- 172.21.15.72:0/3719321558 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3368498771 0 0) 0x7f24b4140c10 con 0x7f24b406fb00 2024-03-29T11:38:43.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.529+0000 7f24b2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:43.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.529+0000 7f24b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:43.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.529+0000 7f24b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:43.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.529+0000 7f24b2ffd640 10 monclient(hunting): my global_id is 4353 2024-03-29T11:38:43.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.529+0000 7f24b2ffd640 1 -- 172.21.15.72:0/3719321558 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24b4072280 con 0x7f24b406fb00 2024-03-29T11:38:43.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.529+0000 7f24b2ffd640 1 -- 172.21.15.72:0/3719321558 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1823478197 0 0) 0x7f249c003520 con 0x7f24b40b0420 2024-03-29T11:38:43.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.529+0000 7f24b2ffd640 1 -- 172.21.15.72:0/3719321558 >> v1:172.21.15.89:6789/0 conn(0x7f24b40af990 legacy=0x7f24b414b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:38:43.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.529+0000 7f24b2ffd640 1 -- 172.21.15.72:0/3719321558 >> v1:172.21.15.72:6789/0 conn(0x7f24b406fb00 legacy=0x7f24b406fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:43.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.529+0000 7f24b2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:38:43.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.529+0000 7f24b2ffd640 1 monclient: found mon.noname-b 2024-03-29T11:38:43.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.529+0000 7f24b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:43.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.529+0000 7f24b2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.72:6790/0 2024-03-29T11:38:43.533 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.530+0000 7f24b2ffd640 1 -- 172.21.15.72:0/3719321558 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b40710a0 con 0x7f24b40b0420 2024-03-29T11:38:43.533 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.530+0000 7f24b2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:38:43.533 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.530+0000 7f24b2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:38:43.533 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.530+0000 7f24b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:43.533 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.530+0000 7f24b9794640 5 monclient: authenticate success, global_id 4361 2024-03-29T11:38:43.533 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.530+0000 7f24b9794640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:43.533 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.530+0000 7f24b2ffd640 1 -- 172.21.15.72:0/3719321558 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f24a4001690 con 0x7f24b40b0420 2024-03-29T11:38:43.533 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.530+0000 7f24b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:43.533 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.530+0000 7f24b2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:43.533 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.530+0000 7f24b2ffd640 10 monclient: dump: 2024-03-29T11:38:43.533 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:38:43.533 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:38:43.534 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:43.534 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:43.534 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:43.534 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:43.534 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:38:43.534 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:38:43.534 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:38:43.534 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:43.534 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.530+0000 7f24b9794640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:43.534 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.531+0000 7f24b2ffd640 1 -- 172.21.15.72:0/3719321558 <== mon.2 v1:172.21.15.72:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24a40017f0 con 0x7f24b40b0420 2024-03-29T11:38:43.534 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.531+0000 7f24b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:43.536 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.531+0000 7f24b9794640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:43.536 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.531+0000 7f24b9794640 4 set_mon_vals no callback set 2024-03-29T11:38:43.536 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.531+0000 7f24b2ffd640 1 -- 172.21.15.72:0/3719321558 <== mon.2 v1:172.21.15.72:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f24a4001bc0 con 0x7f24b40b0420 2024-03-29T11:38:43.536 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.531+0000 7f24b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:43.536 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.531+0000 7f24b2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:43.536 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.531+0000 7f24b2ffd640 10 monclient: dump: 2024-03-29T11:38:43.536 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:38:43.536 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:38:43.536 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:43.536 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:43.537 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:43.537 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:43.537 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:38:43.537 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:38:43.537 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:38:43.537 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:43.537 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.532+0000 7f24b9794640 10 monclient: shutdown 2024-03-29T11:38:43.537 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.532+0000 7f24b9794640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:43.537 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.532+0000 7f24b9794640 1 -- 172.21.15.72:0/3719321558 >> v1:172.21.15.72:6790/0 conn(0x7f24b40b0420 legacy=0x7f24b4140500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:43.537 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.533+0000 7f24b9794640 1 -- 172.21.15.72:0/3719321558 shutdown_connections 2024-03-29T11:38:43.537 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.533+0000 7f24b9794640 1 -- 172.21.15.72:0/3719321558 >> 172.21.15.72:0/3719321558 conn(0x7f24b40672e0 msgr2=0x7f24b40b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:43.537 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.533+0000 7f24b9794640 1 -- 172.21.15.72:0/3719321558 shutdown_connections 2024-03-29T11:38:43.537 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.533+0000 7f24b9794640 1 -- 172.21.15.72:0/3719321558 wait complete. 2024-03-29T11:38:43.537 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.534+0000 7f24b9794640 10 monclient: build_initial_monmap 2024-03-29T11:38:43.537 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.534+0000 7f24b9794640 10 monclient: monmap: 2024-03-29T11:38:43.537 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:38:43.538 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:43.538 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:38:43.538 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:38:43.538 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:43.538 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:43.538 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:38:43.538 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:38:43.538 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:38:43.538 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:43.538 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.534+0000 7f24b9794640 1 librados: starting msgr at 2024-03-29T11:38:43.538 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.534+0000 7f24b9794640 1 librados: starting objecter 2024-03-29T11:38:43.538 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.534+0000 7f24b9794640 1 Processor -- start 2024-03-29T11:38:43.540 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.535+0000 7f24b9794640 1 -- start start 2024-03-29T11:38:43.540 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.535+0000 7f24b9794640 1 librados: setting wanted keys 2024-03-29T11:38:43.540 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.535+0000 7f24b9794640 1 librados: calling monclient init 2024-03-29T11:38:43.540 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.535+0000 7f24b9794640 10 monclient: init 2024-03-29T11:38:43.540 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.537+0000 7f24b9794640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:43.540 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.537+0000 7f24b9794640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:38:43.540 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.537+0000 7f24b9794640 10 monclient(hunting): picked mon.noname-c con 0x7f24b406fb00 addr v1:172.21.15.72:6790/0 2024-03-29T11:38:43.541 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.537+0000 7f24b9794640 10 monclient(hunting): picked mon.noname-a con 0x7f24b40af990 addr v1:172.21.15.72:6789/0 2024-03-29T11:38:43.541 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.537+0000 7f24b9794640 10 monclient(hunting): picked mon.noname-b con 0x7f24b40b0420 addr v1:172.21.15.89:6789/0 2024-03-29T11:38:43.541 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.537+0000 7f24b9794640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24b412ab90 con 0x7f24b40af990 2024-03-29T11:38:43.541 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.537+0000 7f24b9794640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24b413c800 con 0x7f24b40b0420 2024-03-29T11:38:43.541 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.537+0000 7f24b9794640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24b413cae0 con 0x7f24b406fb00 2024-03-29T11:38:43.541 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.537+0000 7f24b9794640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:43.541 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.537+0000 7f24b9794640 10 monclient(hunting): authenticate will time out at 1082.825416s 2024-03-29T11:38:43.541 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.539+0000 7f24b3fff640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f24b406fb00 0x7f24b4141130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:43010/0 (socket says 172.21.15.72:43010) 2024-03-29T11:38:43.541 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.539+0000 7f24b3fff640 1 -- 172.21.15.72:0/1815575310 learned_addr learned my addr 172.21.15.72:0/1815575310 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:38:43.545 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.540+0000 7f24b0ff9640 10 client.?.objecter ms_handle_connect 0x7f24b406fb00 2024-03-29T11:38:43.545 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.540+0000 7f24b0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:43.545 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.541+0000 7f24b0ff9640 10 client.?.objecter ms_handle_connect 0x7f24b40b0420 2024-03-29T11:38:43.545 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.541+0000 7f24b0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:43.545 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.541+0000 7f24b0ff9640 1 -- 172.21.15.72:0/1815575310 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1383943081 0 0) 0x7f24b413cae0 con 0x7f24b406fb00 2024-03-29T11:38:43.545 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.541+0000 7f24b0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:38:43.545 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.541+0000 7f24b0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:43.546 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.541+0000 7f24b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:43.546 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.541+0000 7f24b0ff9640 10 monclient(hunting): my global_id is 4364 2024-03-29T11:38:43.546 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.541+0000 7f24b0ff9640 1 -- 172.21.15.72:0/1815575310 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24a0004eb0 con 0x7f24b406fb00 2024-03-29T11:38:43.546 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.542+0000 7f24b0ff9640 10 client.?.objecter ms_handle_connect 0x7f24b40af990 2024-03-29T11:38:43.546 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.542+0000 7f24b0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:43.546 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.542+0000 7f24b0ff9640 1 -- 172.21.15.72:0/1815575310 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 516764449 0 0) 0x7f24a0004eb0 con 0x7f24b406fb00 2024-03-29T11:38:43.546 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.542+0000 7f24b0ff9640 1 -- 172.21.15.72:0/1815575310 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f24b413cae0 con 0x7f24b406fb00 2024-03-29T11:38:43.546 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.542+0000 7f24b0ff9640 1 -- 172.21.15.72:0/1815575310 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2574373933 0 0) 0x7f24b412ab90 con 0x7f24b40af990 2024-03-29T11:38:43.546 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.543+0000 7f24b0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:38:43.546 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.543+0000 7f24b0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:43.546 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.543+0000 7f24b0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:43.546 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.543+0000 7f24b0ff9640 10 monclient(hunting): my global_id is 4356 2024-03-29T11:38:43.546 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.543+0000 7f24b0ff9640 1 -- 172.21.15.72:0/1815575310 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24a0004eb0 con 0x7f24b40af990 2024-03-29T11:38:43.546 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.543+0000 7f24b0ff9640 1 -- 172.21.15.72:0/1815575310 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f24a8001a10 con 0x7f24b406fb00 2024-03-29T11:38:43.546 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.543+0000 7f24b0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:43.546 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.543+0000 7f24b0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:43.547 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.543+0000 7f24b0ff9640 10 monclient(hunting): dump: 2024-03-29T11:38:43.547 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:38:43.547 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:38:43.547 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:43.547 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:43.547 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:43.547 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:43.547 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:38:43.547 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:38:43.547 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:38:43.547 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:43.547 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.543+0000 7f24b0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:43.547 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.543+0000 7f24b0ff9640 1 -- 172.21.15.72:0/1815575310 <== mon.2 v1:172.21.15.72:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1595560937 0 0) 0x7f24b413cae0 con 0x7f24b406fb00 2024-03-29T11:38:43.547 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.543+0000 7f24b0ff9640 1 -- 172.21.15.72:0/1815575310 >> v1:172.21.15.89:6789/0 conn(0x7f24b40b0420 legacy=0x7f24b413a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:43.547 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.543+0000 7f24b0ff9640 1 -- 172.21.15.72:0/1815575310 >> v1:172.21.15.72:6789/0 conn(0x7f24b40af990 legacy=0x7f24b413b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:43.547 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.543+0000 7f24b0ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:38:43.548 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.543+0000 7f24b0ff9640 1 monclient: found mon.c 2024-03-29T11:38:43.548 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.543+0000 7f24b0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:43.548 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.544+0000 7f24b0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:38:43.548 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.544+0000 7f24b0ff9640 1 -- 172.21.15.72:0/1815575310 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b413b260 con 0x7f24b406fb00 2024-03-29T11:38:43.548 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.544+0000 7f24b0ff9640 10 monclient: _finish_auth 0 2024-03-29T11:38:43.548 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.544+0000 7f24b0ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:38:43.548 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.544+0000 7f24b0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:43.548 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.544+0000 7f24b0ff9640 1 -- 172.21.15.72:0/1815575310 <== mon.2 v1:172.21.15.72:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24a8001d70 con 0x7f24b406fb00 2024-03-29T11:38:43.548 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.544+0000 7f24b0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:43.548 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.544+0000 7f24b0ff9640 1 -- 172.21.15.72:0/1815575310 <== mon.2 v1:172.21.15.72:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f24a8003a60 con 0x7f24b406fb00 2024-03-29T11:38:43.548 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.544+0000 7f24b0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:43.548 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.544+0000 7f24b0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:43.548 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.544+0000 7f24b0ff9640 10 monclient: dump: 2024-03-29T11:38:43.548 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:38:43.548 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:38:43.548 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:43.548 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:43.548 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:43.549 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:43.549 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:38:43.549 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:38:43.549 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:38:43.549 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:43.551 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.545+0000 7f24b9794640 5 monclient: authenticate success, global_id 4364 2024-03-29T11:38:43.551 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.545+0000 7f24b9794640 10 monclient: _renew_subs 2024-03-29T11:38:43.551 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.545+0000 7f24b9794640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:38:43.551 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.545+0000 7f24b9794640 1 -- 172.21.15.72:0/1815575310 --> v1:172.21.15.72:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24b41392d0 con 0x7f24b406fb00 2024-03-29T11:38:43.551 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.545+0000 7f24b27fc640 4 set_mon_vals no callback set 2024-03-29T11:38:43.551 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.545+0000 7f24b9794640 10 client.4364.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:43.551 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.545+0000 7f24b9794640 10 monclient: _renew_subs 2024-03-29T11:38:43.551 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.545+0000 7f24b9794640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:38:43.551 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.545+0000 7f24b9794640 1 -- 172.21.15.72:0/1815575310 --> v1:172.21.15.72:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f24b41e7410 con 0x7f24b406fb00 2024-03-29T11:38:43.551 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.545+0000 7f24b9794640 1 librados: init done 2024-03-29T11:38:43.551 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.546+0000 7f24b9794640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:43.551 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.546+0000 7f24b9794640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:43.551 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.546+0000 7f24b9794640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:38:43.551 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.546+0000 7f24b9794640 1 -- 172.21.15.72:0/1815575310 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24b404f730 con 0x7f24b406fb00 2024-03-29T11:38:43.551 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.548+0000 7f24b3fff640 1 -- 172.21.15.72:0/1815575310 >> v1:172.21.15.72:6790/0 conn(0x7f24b406fb00 legacy=0x7f24b4141130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-29T11:38:43.552 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.548+0000 7f24b3fff640 1 -- 172.21.15.72:0/1815575310 >> v1:172.21.15.72:6790/0 conn(0x7f24b406fb00 legacy=0x7f24b4141130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:38:43.552 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.548+0000 7f24b3fff640 1 --1- 172.21.15.72:0/1815575310 >> v1:172.21.15.72:6790/0 conn(0x7f24b406fb00 0x7f24b4141130 :-1 s=OPENED pgs=373924 cs=1 l=1).handle_message read tag failed 2024-03-29T11:38:43.552 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.548+0000 7f24b3fff640 1 --1- 172.21.15.72:0/1815575310 >> v1:172.21.15.72:6790/0 conn(0x7f24b406fb00 0x7f24b4141130 :-1 s=OPENED pgs=373924 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:38:43.552 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.548+0000 7f24b0ff9640 10 monclient: ms_handle_reset current mon v1:172.21.15.72:6790/0 2024-03-29T11:38:43.552 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.548+0000 7f24b0ff9640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:43.552 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.548+0000 7f24b0ff9640 1 -- 172.21.15.72:0/1815575310 >> v1:172.21.15.72:6790/0 conn(0x7f24b406fb00 legacy=0x7f24b4141130 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:38:43.552 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.548+0000 7f24b0ff9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:38:43.552 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.548+0000 7f24b0ff9640 10 monclient(hunting): picked mon.c con 0x7f24a00062c0 addr v1:172.21.15.72:6790/0 2024-03-29T11:38:43.552 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.549+0000 7f24b0ff9640 10 monclient(hunting): picked mon.b con 0x7f24b40b0420 addr v1:172.21.15.89:6789/0 2024-03-29T11:38:43.552 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.549+0000 7f24b0ff9640 10 monclient(hunting): picked mon.a con 0x7f24b40af990 addr v1:172.21.15.72:6789/0 2024-03-29T11:38:43.552 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.549+0000 7f24b0ff9640 1 -- 172.21.15.72:0/1815575310 --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f24b413cae0 con 0x7f24b40af990 2024-03-29T11:38:43.552 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.549+0000 7f24b0ff9640 1 -- 172.21.15.72:0/1815575310 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f24b412ab90 con 0x7f24b40b0420 2024-03-29T11:38:43.552 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.549+0000 7f24b0ff9640 1 -- 172.21.15.72:0/1815575310 --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f24a0013a90 con 0x7f24a00062c0 2024-03-29T11:38:43.552 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.549+0000 7f24b0ff9640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:43.553 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.552+0000 7f24b0ff9640 10 client.4364.objecter ms_handle_connect 0x7f24a00062c0 2024-03-29T11:38:43.553 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.552+0000 7f24b0ff9640 10 client.4364.objecter resend_mon_ops 2024-03-29T11:38:43.554 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.553+0000 7f24b0ff9640 10 client.4364.objecter ms_handle_connect 0x7f24b40af990 2024-03-29T11:38:43.555 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.553+0000 7f24b0ff9640 10 client.4364.objecter resend_mon_ops 2024-03-29T11:38:43.555 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.553+0000 7f24b0ff9640 1 -- 172.21.15.72:0/1815575310 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2499618699 0 0) 0x7f24a0013a90 con 0x7f24a00062c0 2024-03-29T11:38:43.555 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.553+0000 7f24b0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:38:43.555 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.553+0000 7f24b0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:43.555 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.553+0000 7f24b0ff9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T11:38:43.555 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.553+0000 7f24b0ff9640 1 -- 172.21.15.72:0/1815575310 --> v1:172.21.15.72:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f24a0012a10 con 0x7f24a00062c0 2024-03-29T11:38:43.555 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.554+0000 7f24b0ff9640 10 client.4364.objecter ms_handle_connect 0x7f24b40b0420 2024-03-29T11:38:43.555 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.554+0000 7f24b0ff9640 10 client.4364.objecter resend_mon_ops 2024-03-29T11:38:43.555 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.554+0000 7f24b0ff9640 1 -- 172.21.15.72:0/1815575310 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1772871743 0 0) 0x7f24b412ab90 con 0x7f24b40b0420 2024-03-29T11:38:43.555 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.554+0000 7f24b0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:38:43.555 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.554+0000 7f24b0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:43.555 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.554+0000 7f24b0ff9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T11:38:43.555 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.554+0000 7f24b0ff9640 1 -- 172.21.15.72:0/1815575310 --> v1:172.21.15.89:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f24a0013a90 con 0x7f24b40b0420 2024-03-29T11:38:43.558 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.554+0000 7f24b0ff9640 1 -- 172.21.15.72:0/1815575310 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 2789034339 0 0) 0x7f24a0013a90 con 0x7f24b40b0420 2024-03-29T11:38:43.558 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.554+0000 7f24b0ff9640 1 -- 172.21.15.72:0/1815575310 >> v1:172.21.15.72:6790/0 conn(0x7f24a00062c0 legacy=0x7f24a00066a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:43.558 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.554+0000 7f24b0ff9640 1 -- 172.21.15.72:0/1815575310 >> v1:172.21.15.72:6789/0 conn(0x7f24b40af990 legacy=0x7f24a000fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:43.559 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.554+0000 7f24b0ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:38:43.559 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.554+0000 7f24b0ff9640 1 monclient: found mon.b 2024-03-29T11:38:43.559 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.554+0000 7f24b0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:43.559 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.554+0000 7f24b0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:38:43.559 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.554+0000 7f24b0ff9640 1 -- 172.21.15.72:0/1815575310 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f24a0014e70 con 0x7f24b40b0420 2024-03-29T11:38:43.559 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.554+0000 7f24b0ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:43.559 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.554+0000 7f24b0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:38:43.559 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.554+0000 7f24b0ff9640 1 -- 172.21.15.72:0/1815575310 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24b412ab90 con 0x7f24b40b0420 2024-03-29T11:38:43.559 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.554+0000 7f24b0ff9640 10 monclient: _finish_auth 0 2024-03-29T11:38:43.559 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.554+0000 7f24b0ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:38:43.559 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.554+0000 7f24b0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:43.559 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.554+0000 7f24b0ff9640 1 -- 172.21.15.72:0/1815575310 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f24a40015e0 con 0x7f24b40b0420 2024-03-29T11:38:43.559 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.554+0000 7f24b0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:43.559 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.554+0000 7f24b0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:43.559 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.554+0000 7f24b0ff9640 10 monclient: dump: 2024-03-29T11:38:43.560 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:38:43.560 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:38:43.560 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:43.560 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:43.560 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:43.560 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:43.560 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:38:43.560 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:38:43.560 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:38:43.560 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:43.560 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.555+0000 7f24b0ff9640 1 -- 172.21.15.72:0/1815575310 <== mon.1 v1:172.21.15.89:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24a40019a0 con 0x7f24b40b0420 2024-03-29T11:38:43.560 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.555+0000 7f24b0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:43.560 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.555+0000 7f24b1ffb640 4 set_mon_vals no callback set 2024-03-29T11:38:43.560 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.556+0000 7f24b0ff9640 1 -- 172.21.15.72:0/1815575310 <== mon.1 v1:172.21.15.89:6789/0 5 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f24a4001b90 con 0x7f24b40b0420 2024-03-29T11:38:43.560 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.557+0000 7f24b0ff9640 1 -- 172.21.15.72:0/1815575310 <== mon.1 v1:172.21.15.89:6789/0 6 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f24a40841a0 con 0x7f24b40b0420 2024-03-29T11:38:43.560 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.557+0000 7f24b0ff9640 10 client.4364.objecter ms_dispatch 0x7f24b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:43.561 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.557+0000 7f24b0ff9640 3 client.4364.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:43.561 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.557+0000 7f24b0ff9640 3 client.4364.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:43.561 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.557+0000 7f24b0ff9640 20 client.4364.objecter dump_active .. 0 homeless 2024-03-29T11:38:43.561 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.557+0000 7f24b3fff640 10 monclient: get_auth_request con 0x7f24b406fb00 auth_method 0 2024-03-29T11:38:43.561 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.558+0000 7f24b0ff9640 10 client.4364.objecter ms_handle_connect 0x7f24b406fb00 2024-03-29T11:38:43.564 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.563+0000 7f24b0ff9640 1 -- 172.21.15.72:0/1815575310 <== mon.1 v1:172.21.15.89:6789/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f24a404f8b0 con 0x7f24b40b0420 2024-03-29T11:38:43.564 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.563+0000 7f24b0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:43.564 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.563+0000 7f24b0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:43.882 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.881+0000 7f24b9794640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:43.882 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.881+0000 7f24b9794640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:43.882 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.881+0000 7f24b9794640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:38:43.882 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.881+0000 7f24b9794640 1 -- 172.21.15.72:0/1815575310 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24b4140080 con 0x7f24b40b0420 2024-03-29T11:38:43.883 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.883+0000 7f24b0ff9640 1 -- 172.21.15.72:0/1815575310 <== mon.1 v1:172.21.15.89:6789/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f24a40519b0 con 0x7f24b40b0420 2024-03-29T11:38:43.883 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.883+0000 7f24b0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:43.883 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.883+0000 7f24b0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:43.885 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:38:43.885 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:43.890 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.887+0000 7f24b9794640 10 librados: watch_flush enter 2024-03-29T11:38:43.890 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.887+0000 7f24b9794640 10 librados: watch_flush exit 2024-03-29T11:38:43.890 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.887+0000 7f24b9794640 20 client.4364.objecter shutdown clearing up homeless session... 2024-03-29T11:38:43.890 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.887+0000 7f24b9794640 10 client.4364.objecter successfully canceled tick 2024-03-29T11:38:43.890 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.887+0000 7f24b9794640 1 -- 172.21.15.72:0/1815575310 >> v1:172.21.15.72:6816/1578866102 conn(0x7f24b406fb00 legacy=0x7f24a007bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:43.890 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.887+0000 7f24b9794640 10 monclient: shutdown 2024-03-29T11:38:43.890 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.887+0000 7f24b9794640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:43.891 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.887+0000 7f24b9794640 1 -- 172.21.15.72:0/1815575310 >> v1:172.21.15.89:6789/0 conn(0x7f24b40b0420 legacy=0x7f24a000c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:43.891 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.887+0000 7f24b9794640 1 -- 172.21.15.72:0/1815575310 shutdown_connections 2024-03-29T11:38:43.891 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.887+0000 7f24b9794640 1 -- 172.21.15.72:0/1815575310 >> 172.21.15.72:0/1815575310 conn(0x7f24b40672e0 msgr2=0x7f24b4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:43.891 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.889+0000 7f24b9794640 1 -- 172.21.15.72:0/1815575310 shutdown_connections 2024-03-29T11:38:43.892 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.892+0000 7f24b9794640 1 -- 172.21.15.72:0/1815575310 wait complete. 2024-03-29T11:38:43.893 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:43.892+0000 7f24b9794640 1 librados: shutdown 2024-03-29T11:38:43.910 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:50.716 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:50.872 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.869+0000 7fd925e14640 10 monclient: get_monmap_and_config 2024-03-29T11:38:50.872 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.869+0000 7fd925e14640 10 monclient: build_initial_monmap 2024-03-29T11:38:50.873 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.869+0000 7fd925e14640 10 monclient: monmap: 2024-03-29T11:38:50.873 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:38:50.873 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:38:50.873 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:38:50.871304+0000 2024-03-29T11:38:50.873 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:38:50.871304+0000 2024-03-29T11:38:50.873 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:50.873 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:50.873 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:38:50.873 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:38:50.873 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:38:50.874 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:50.874 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.870+0000 7fd925e14640 1 Processor -- start 2024-03-29T11:38:50.874 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.871+0000 7fd925e14640 1 -- start start 2024-03-29T11:38:50.878 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.871+0000 7fd925e14640 10 monclient: init 2024-03-29T11:38:50.878 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.871+0000 7fd925e14640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:50.878 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.871+0000 7fd925e14640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:38:50.878 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.871+0000 7fd925e14640 10 monclient(hunting): picked mon.noname-a con 0x7fd9200a9ac0 addr v1:172.21.15.72:6789/0 2024-03-29T11:38:50.878 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.871+0000 7fd925e14640 10 monclient(hunting): picked mon.noname-b con 0x7fd92013ee90 addr v1:172.21.15.72:6790/0 2024-03-29T11:38:50.878 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.871+0000 7fd925e14640 10 monclient(hunting): picked mon.noname-c con 0x7fd9200af900 addr v1:172.21.15.89:6789/0 2024-03-29T11:38:50.878 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.871+0000 7fd925e14640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9200afd10 con 0x7fd9200a9ac0 2024-03-29T11:38:50.878 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.872+0000 7fd925e14640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9200af6f0 con 0x7fd9200af900 2024-03-29T11:38:50.878 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.872+0000 7fd925e14640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9200aac80 con 0x7fd92013ee90 2024-03-29T11:38:50.878 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.872+0000 7fd925e14640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:50.879 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.872+0000 7fd925e14640 10 monclient(hunting): authenticate will time out at 1090.161027s 2024-03-29T11:38:50.879 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.876+0000 7fd91effd640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7fd92013ee90 0x7fd92013b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:57653/0 (socket says 172.21.15.72:57653) 2024-03-29T11:38:50.879 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.876+0000 7fd91effd640 1 -- 172.21.15.72:0/3195646165 learned_addr learned my addr 172.21.15.72:0/3195646165 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:38:50.879 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.876+0000 7fd91e7fc640 1 -- 172.21.15.72:0/3195646165 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3770324059 0 0) 0x7fd9200aac80 con 0x7fd92013ee90 2024-03-29T11:38:50.879 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.876+0000 7fd91e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:38:50.879 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.876+0000 7fd91e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:50.879 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.876+0000 7fd91e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:50.879 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.876+0000 7fd91e7fc640 10 monclient(hunting): my global_id is 4367 2024-03-29T11:38:50.879 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.876+0000 7fd91e7fc640 1 -- 172.21.15.72:0/3195646165 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd908003520 con 0x7fd92013ee90 2024-03-29T11:38:50.882 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.878+0000 7fd91e7fc640 1 -- 172.21.15.72:0/3195646165 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2242471795 0 0) 0x7fd908003520 con 0x7fd92013ee90 2024-03-29T11:38:50.882 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.878+0000 7fd91e7fc640 1 -- 172.21.15.72:0/3195646165 >> v1:172.21.15.89:6789/0 conn(0x7fd9200af900 legacy=0x7fd920149b00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:38:50.882 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.878+0000 7fd91e7fc640 1 -- 172.21.15.72:0/3195646165 >> v1:172.21.15.72:6789/0 conn(0x7fd9200a9ac0 legacy=0x7fd9200adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:50.882 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.878+0000 7fd91e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:38:50.882 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.878+0000 7fd91e7fc640 1 monclient: found mon.noname-b 2024-03-29T11:38:50.882 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.878+0000 7fd91e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:50.882 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.878+0000 7fd91e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.72:6790/0 2024-03-29T11:38:50.882 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.878+0000 7fd91e7fc640 1 -- 172.21.15.72:0/3195646165 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9200aaf60 con 0x7fd92013ee90 2024-03-29T11:38:50.883 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.878+0000 7fd91e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:38:50.883 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.878+0000 7fd91e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:38:50.883 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.878+0000 7fd91e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:50.883 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.878+0000 7fd925e14640 5 monclient: authenticate success, global_id 4367 2024-03-29T11:38:50.883 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.878+0000 7fd91e7fc640 1 -- 172.21.15.72:0/3195646165 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fd910001690 con 0x7fd92013ee90 2024-03-29T11:38:50.883 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.878+0000 7fd925e14640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:50.883 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.878+0000 7fd91e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:50.883 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.878+0000 7fd91e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:50.883 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.878+0000 7fd91e7fc640 10 monclient: dump: 2024-03-29T11:38:50.883 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:38:50.883 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:38:50.883 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:50.883 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:50.883 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:50.883 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:50.883 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:38:50.884 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:38:50.884 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:38:50.884 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:50.884 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.878+0000 7fd925e14640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:50.884 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.878+0000 7fd91e7fc640 1 -- 172.21.15.72:0/3195646165 <== mon.2 v1:172.21.15.72:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd910001860 con 0x7fd92013ee90 2024-03-29T11:38:50.884 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.878+0000 7fd91e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:50.884 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.878+0000 7fd925e14640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:50.884 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.878+0000 7fd925e14640 4 set_mon_vals no callback set 2024-03-29T11:38:50.884 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.879+0000 7fd925e14640 10 monclient: shutdown 2024-03-29T11:38:50.884 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.879+0000 7fd925e14640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:50.884 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.879+0000 7fd925e14640 1 -- 172.21.15.72:0/3195646165 >> v1:172.21.15.72:6790/0 conn(0x7fd92013ee90 legacy=0x7fd92013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:50.884 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.879+0000 7fd925e14640 1 -- 172.21.15.72:0/3195646165 shutdown_connections 2024-03-29T11:38:50.884 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.879+0000 7fd925e14640 1 -- 172.21.15.72:0/3195646165 >> 172.21.15.72:0/3195646165 conn(0x7fd9200b35e0 msgr2=0x7fd9200b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:50.884 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.879+0000 7fd925e14640 1 -- 172.21.15.72:0/3195646165 shutdown_connections 2024-03-29T11:38:50.885 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.883+0000 7fd925e14640 1 -- 172.21.15.72:0/3195646165 wait complete. 2024-03-29T11:38:50.886 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.884+0000 7fd925e14640 10 monclient: build_initial_monmap 2024-03-29T11:38:50.886 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.884+0000 7fd925e14640 10 monclient: monmap: 2024-03-29T11:38:50.886 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:38:50.886 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:50.886 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:38:50.886 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:38:50.886 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:50.886 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:50.886 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:38:50.886 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:38:50.887 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:38:50.887 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:50.888 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.885+0000 7fd925e14640 1 librados: starting msgr at 2024-03-29T11:38:50.888 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.885+0000 7fd925e14640 1 librados: starting objecter 2024-03-29T11:38:50.888 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.885+0000 7fd925e14640 1 Processor -- start 2024-03-29T11:38:50.888 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.886+0000 7fd925e14640 1 -- start start 2024-03-29T11:38:50.888 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.886+0000 7fd925e14640 1 librados: setting wanted keys 2024-03-29T11:38:50.888 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.886+0000 7fd925e14640 1 librados: calling monclient init 2024-03-29T11:38:50.888 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.886+0000 7fd925e14640 10 monclient: init 2024-03-29T11:38:50.889 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.886+0000 7fd925e14640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:50.889 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.886+0000 7fd925e14640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:38:50.889 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.886+0000 7fd925e14640 10 monclient(hunting): picked mon.noname-b con 0x7fd9200a9ac0 addr v1:172.21.15.89:6789/0 2024-03-29T11:38:50.889 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.886+0000 7fd925e14640 10 monclient(hunting): picked mon.noname-a con 0x7fd9200af900 addr v1:172.21.15.72:6789/0 2024-03-29T11:38:50.889 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.886+0000 7fd925e14640 10 monclient(hunting): picked mon.noname-c con 0x7fd92013ee90 addr v1:172.21.15.72:6790/0 2024-03-29T11:38:50.890 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.886+0000 7fd925e14640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9201d71f0 con 0x7fd9200af900 2024-03-29T11:38:50.890 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.886+0000 7fd925e14640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd920072410 con 0x7fd9200a9ac0 2024-03-29T11:38:50.890 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.886+0000 7fd925e14640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9200725a0 con 0x7fd92013ee90 2024-03-29T11:38:50.890 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.886+0000 7fd925e14640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:50.890 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.886+0000 7fd925e14640 10 monclient(hunting): authenticate will time out at 1090.175984s 2024-03-29T11:38:50.892 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.890+0000 7fd91effd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fd9200af900 0x7fd920136730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50905/0 (socket says 172.21.15.72:50905) 2024-03-29T11:38:50.892 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.890+0000 7fd91effd640 1 -- 172.21.15.72:0/2273934895 learned_addr learned my addr 172.21.15.72:0/2273934895 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:38:50.892 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.890+0000 7fd907fff640 10 client.?.objecter ms_handle_connect 0x7fd9200af900 2024-03-29T11:38:50.892 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.890+0000 7fd907fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:50.892 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.890+0000 7fd907fff640 1 -- 172.21.15.72:0/2273934895 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 114198587 0 0) 0x7fd9201d71f0 con 0x7fd9200af900 2024-03-29T11:38:50.892 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.890+0000 7fd907fff640 10 monclient(hunting): _negotiate 2024-03-29T11:38:50.893 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.890+0000 7fd907fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:50.893 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.890+0000 7fd907fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:50.893 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.890+0000 7fd907fff640 10 monclient(hunting): my global_id is 4362 2024-03-29T11:38:50.893 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.890+0000 7fd907fff640 1 -- 172.21.15.72:0/2273934895 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd90c004eb0 con 0x7fd9200af900 2024-03-29T11:38:50.894 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.891+0000 7fd907fff640 1 -- 172.21.15.72:0/2273934895 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1809710210 0 0) 0x7fd90c004eb0 con 0x7fd9200af900 2024-03-29T11:38:50.894 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.891+0000 7fd907fff640 1 -- 172.21.15.72:0/2273934895 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9201d71f0 con 0x7fd9200af900 2024-03-29T11:38:50.894 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.891+0000 7fd907fff640 1 -- 172.21.15.72:0/2273934895 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fd9100049c0 con 0x7fd9200af900 2024-03-29T11:38:50.894 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.891+0000 7fd907fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:50.894 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.891+0000 7fd907fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:38:50.894 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.891+0000 7fd907fff640 10 monclient(hunting): dump: 2024-03-29T11:38:50.894 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:38:50.894 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:38:50.894 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:50.894 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:50.895 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:50.895 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:50.895 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:38:50.895 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:38:50.895 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:38:50.895 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:50.895 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.891+0000 7fd907fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:50.896 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.893+0000 7fd907fff640 1 -- 172.21.15.72:0/2273934895 <== mon.0 v1:172.21.15.72:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4084521191 0 0) 0x7fd9201d71f0 con 0x7fd9200af900 2024-03-29T11:38:50.896 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.893+0000 7fd907fff640 1 -- 172.21.15.72:0/2273934895 >> v1:172.21.15.72:6790/0 conn(0x7fd92013ee90 legacy=0x7fd9201405e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:50.896 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.893+0000 7fd907fff640 1 -- 172.21.15.72:0/2273934895 >> v1:172.21.15.89:6789/0 conn(0x7fd9200a9ac0 legacy=0x7fd9201d6530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:38:50.896 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.893+0000 7fd907fff640 10 monclient: _finish_hunting 0 2024-03-29T11:38:50.896 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.893+0000 7fd907fff640 1 monclient: found mon.a 2024-03-29T11:38:50.896 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.893+0000 7fd907fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:50.896 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.893+0000 7fd907fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:38:50.896 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.893+0000 7fd907fff640 1 -- 172.21.15.72:0/2273934895 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9201d6c40 con 0x7fd9200af900 2024-03-29T11:38:50.896 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.893+0000 7fd907fff640 10 monclient: _finish_auth 0 2024-03-29T11:38:50.896 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.894+0000 7fd907fff640 10 monclient: _check_auth_tickets 2024-03-29T11:38:50.896 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.894+0000 7fd907fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:50.897 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.894+0000 7fd925e14640 5 monclient: authenticate success, global_id 4362 2024-03-29T11:38:50.897 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.894+0000 7fd925e14640 10 monclient: _renew_subs 2024-03-29T11:38:50.897 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.894+0000 7fd925e14640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:38:50.897 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.894+0000 7fd925e14640 1 -- 172.21.15.72:0/2273934895 --> v1:172.21.15.72:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9201d6ef0 con 0x7fd9200af900 2024-03-29T11:38:50.897 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.894+0000 7fd925e14640 10 client.4362.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:50.897 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.894+0000 7fd925e14640 10 monclient: _renew_subs 2024-03-29T11:38:50.897 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.894+0000 7fd925e14640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:38:50.898 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.894+0000 7fd925e14640 1 -- 172.21.15.72:0/2273934895 --> v1:172.21.15.72:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd920137020 con 0x7fd9200af900 2024-03-29T11:38:50.898 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.894+0000 7fd925e14640 1 librados: init done 2024-03-29T11:38:50.898 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.894+0000 7fd907fff640 1 -- 172.21.15.72:0/2273934895 <== mon.0 v1:172.21.15.72:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd910001b10 con 0x7fd9200af900 2024-03-29T11:38:50.898 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.894+0000 7fd907fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:50.898 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.894+0000 7fd91dffb640 4 set_mon_vals no callback set 2024-03-29T11:38:50.898 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.895+0000 7fd907fff640 1 -- 172.21.15.72:0/2273934895 <== mon.0 v1:172.21.15.72:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fd910005030 con 0x7fd9200af900 2024-03-29T11:38:50.898 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.895+0000 7fd907fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:50.898 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.895+0000 7fd907fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:50.898 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.895+0000 7fd907fff640 10 monclient: dump: 2024-03-29T11:38:50.898 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:38:50.898 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:38:50.898 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:50.898 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:50.898 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:50.898 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:50.898 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:38:50.899 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:38:50.899 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:38:50.899 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:50.899 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.897+0000 7fd907fff640 1 -- 172.21.15.72:0/2273934895 <== mon.0 v1:172.21.15.72:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7fd910005190 con 0x7fd9200af900 2024-03-29T11:38:50.900 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.897+0000 7fd925e14640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:50.900 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.897+0000 7fd925e14640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:50.900 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.897+0000 7fd925e14640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:38:50.900 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.897+0000 7fd925e14640 1 -- 172.21.15.72:0/2273934895 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd900005080 con 0x7fd9200af900 2024-03-29T11:38:50.901 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.898+0000 7fd907fff640 1 -- 172.21.15.72:0/2273934895 <== mon.0 v1:172.21.15.72:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7fd910084dc0 con 0x7fd9200af900 2024-03-29T11:38:50.901 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.898+0000 7fd907fff640 10 client.4362.objecter ms_dispatch 0x7fd920000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:50.901 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.898+0000 7fd907fff640 3 client.4362.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:50.901 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.898+0000 7fd907fff640 3 client.4362.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:50.902 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.898+0000 7fd907fff640 20 client.4362.objecter dump_active .. 0 homeless 2024-03-29T11:38:50.902 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.898+0000 7fd91f7fe640 10 monclient: get_auth_request con 0x7fd90c06dc10 auth_method 0 2024-03-29T11:38:50.902 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.899+0000 7fd907fff640 10 client.4362.objecter ms_handle_connect 0x7fd90c06dc10 2024-03-29T11:38:50.910 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.906+0000 7fd907fff640 1 -- 172.21.15.72:0/2273934895 <== mon.0 v1:172.21.15.72:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd9100506a0 con 0x7fd9200af900 2024-03-29T11:38:50.910 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.906+0000 7fd907fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:50.910 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:50.906+0000 7fd907fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:51.239 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:51.237+0000 7fd925e14640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:51.239 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:51.237+0000 7fd925e14640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:51.239 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:51.237+0000 7fd925e14640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:38:51.239 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:51.237+0000 7fd925e14640 1 -- 172.21.15.72:0/2273934895 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd900005330 con 0x7fd9200af900 2024-03-29T11:38:51.240 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:51.238+0000 7fd907fff640 1 -- 172.21.15.72:0/2273934895 <== mon.0 v1:172.21.15.72:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7fd9100525d0 con 0x7fd9200af900 2024-03-29T11:38:51.240 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:51.238+0000 7fd907fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:51.240 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:51.238+0000 7fd907fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:51.242 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:38:51.242 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:51.243 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:51.241+0000 7fd925e14640 10 librados: watch_flush enter 2024-03-29T11:38:51.243 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:51.242+0000 7fd925e14640 10 librados: watch_flush exit 2024-03-29T11:38:51.243 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:51.242+0000 7fd925e14640 20 client.4362.objecter shutdown clearing up homeless session... 2024-03-29T11:38:51.243 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:51.242+0000 7fd925e14640 10 client.4362.objecter successfully canceled tick 2024-03-29T11:38:51.246 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:51.244+0000 7fd925e14640 1 -- 172.21.15.72:0/2273934895 >> v1:172.21.15.72:6816/1578866102 conn(0x7fd90c06dc10 legacy=0x7fd90c0700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:51.246 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:51.244+0000 7fd925e14640 10 monclient: shutdown 2024-03-29T11:38:51.246 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:51.244+0000 7fd925e14640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:51.246 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:51.244+0000 7fd925e14640 1 -- 172.21.15.72:0/2273934895 >> v1:172.21.15.72:6789/0 conn(0x7fd9200af900 legacy=0x7fd920136730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:51.246 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:51.244+0000 7fd925e14640 1 -- 172.21.15.72:0/2273934895 shutdown_connections 2024-03-29T11:38:51.246 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:51.244+0000 7fd925e14640 1 -- 172.21.15.72:0/2273934895 >> 172.21.15.72:0/2273934895 conn(0x7fd9200b35e0 msgr2=0x7fd92013b190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:51.249 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:51.246+0000 7fd925e14640 1 -- 172.21.15.72:0/2273934895 shutdown_connections 2024-03-29T11:38:51.249 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:51.247+0000 7fd925e14640 1 -- 172.21.15.72:0/2273934895 wait complete. 2024-03-29T11:38:51.249 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:51.248+0000 7fd925e14640 1 librados: shutdown 2024-03-29T11:38:51.269 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:58.076 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:58.224 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.220+0000 7f4582e1e640 10 monclient: get_monmap_and_config 2024-03-29T11:38:58.224 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.220+0000 7f4582e1e640 10 monclient: build_initial_monmap 2024-03-29T11:38:58.224 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.220+0000 7f4582e1e640 10 monclient: monmap: 2024-03-29T11:38:58.224 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:38:58.224 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:38:58.224 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:38:58.221414+0000 2024-03-29T11:38:58.224 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:38:58.221414+0000 2024-03-29T11:38:58.224 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:58.225 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:58.225 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:38:58.225 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:38:58.225 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:38:58.225 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:58.225 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.221+0000 7f4582e1e640 1 Processor -- start 2024-03-29T11:38:58.225 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.221+0000 7f4582e1e640 1 -- start start 2024-03-29T11:38:58.225 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.221+0000 7f4582e1e640 10 monclient: init 2024-03-29T11:38:58.225 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.222+0000 7f4582e1e640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:58.225 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.222+0000 7f4582e1e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:38:58.225 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.222+0000 7f4582e1e640 10 monclient(hunting): picked mon.noname-c con 0x7f457c0af6c0 addr v1:172.21.15.89:6789/0 2024-03-29T11:38:58.225 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.222+0000 7f4582e1e640 10 monclient(hunting): picked mon.noname-b con 0x7f457c14f6b0 addr v1:172.21.15.72:6790/0 2024-03-29T11:38:58.225 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.222+0000 7f4582e1e640 10 monclient(hunting): picked mon.noname-a con 0x7f457c152980 addr v1:172.21.15.72:6789/0 2024-03-29T11:38:58.225 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.222+0000 7f4582e1e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f457c0a94e0 con 0x7f457c152980 2024-03-29T11:38:58.225 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.222+0000 7f4582e1e640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f457c0afb60 con 0x7f457c0af6c0 2024-03-29T11:38:58.225 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.222+0000 7f4582e1e640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f457c0aab80 con 0x7f457c14f6b0 2024-03-29T11:38:58.225 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.222+0000 7f4582e1e640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:58.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.222+0000 7f4582e1e640 10 monclient(hunting): authenticate will time out at 1097.510658s 2024-03-29T11:38:58.227 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.225+0000 7f4580bba640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f457c0af6c0 0x7f457c0ad450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:53407/0 (socket says 172.21.15.72:53407) 2024-03-29T11:38:58.227 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.225+0000 7f4580bba640 1 -- 172.21.15.72:0/2188428050 learned_addr learned my addr 172.21.15.72:0/2188428050 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:38:58.228 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.227+0000 7f457b7fe640 1 -- 172.21.15.72:0/2188428050 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4241731104 0 0) 0x7f457c0afb60 con 0x7f457c0af6c0 2024-03-29T11:38:58.228 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.227+0000 7f457b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:38:58.228 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.227+0000 7f457b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:58.228 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.227+0000 7f457b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:58.228 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.227+0000 7f457b7fe640 10 monclient(hunting): my global_id is 4351 2024-03-29T11:38:58.228 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.227+0000 7f457b7fe640 1 -- 172.21.15.72:0/2188428050 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4564003520 con 0x7f457c0af6c0 2024-03-29T11:38:58.228 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.227+0000 7f457b7fe640 1 -- 172.21.15.72:0/2188428050 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1671620830 0 0) 0x7f457c0a94e0 con 0x7f457c152980 2024-03-29T11:38:58.228 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.227+0000 7f457b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:38:58.229 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.227+0000 7f457b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:58.229 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.227+0000 7f457b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:58.229 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.227+0000 7f457b7fe640 10 monclient(hunting): my global_id is 4365 2024-03-29T11:38:58.229 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.227+0000 7f457b7fe640 1 -- 172.21.15.72:0/2188428050 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f457c0afb60 con 0x7f457c152980 2024-03-29T11:38:58.230 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.228+0000 7f457b7fe640 1 -- 172.21.15.72:0/2188428050 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1016525592 0 0) 0x7f457c0afb60 con 0x7f457c152980 2024-03-29T11:38:58.230 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.228+0000 7f457b7fe640 1 -- 172.21.15.72:0/2188428050 >> v1:172.21.15.72:6790/0 conn(0x7f457c14f6b0 legacy=0x7f457c14fae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:38:58.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.228+0000 7f457b7fe640 1 -- 172.21.15.72:0/2188428050 >> v1:172.21.15.89:6789/0 conn(0x7f457c0af6c0 legacy=0x7f457c0ad450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:58.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.228+0000 7f457b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:38:58.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.228+0000 7f457b7fe640 1 monclient: found mon.noname-a 2024-03-29T11:38:58.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.228+0000 7f457b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:58.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.228+0000 7f457b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.72:6789/0 2024-03-29T11:38:58.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.228+0000 7f457b7fe640 1 -- 172.21.15.72:0/2188428050 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f457c0aa680 con 0x7f457c152980 2024-03-29T11:38:58.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.228+0000 7f457b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:38:58.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.228+0000 7f457b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:38:58.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.228+0000 7f457b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:58.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.228+0000 7f457b7fe640 1 -- 172.21.15.72:0/2188428050 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f4570001520 con 0x7f457c152980 2024-03-29T11:38:58.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.228+0000 7f457b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:58.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.228+0000 7f457b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:38:58.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.228+0000 7f457b7fe640 10 monclient: dump: 2024-03-29T11:38:58.231 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:38:58.231 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:38:58.231 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:58.232 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:58.232 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:58.232 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:58.232 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:38:58.232 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:38:58.232 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:38:58.232 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:58.232 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.229+0000 7f4582e1e640 5 monclient: authenticate success, global_id 4365 2024-03-29T11:38:58.232 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.229+0000 7f4582e1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:58.232 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.229+0000 7f457b7fe640 1 -- 172.21.15.72:0/2188428050 <== mon.0 v1:172.21.15.72:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4570001680 con 0x7f457c152980 2024-03-29T11:38:58.232 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.229+0000 7f457b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:58.232 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.229+0000 7f457b7fe640 1 -- 172.21.15.72:0/2188428050 <== mon.0 v1:172.21.15.72:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f4570001ad0 con 0x7f457c152980 2024-03-29T11:38:58.232 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.229+0000 7f457b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:58.232 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.229+0000 7f457b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:58.232 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.229+0000 7f457b7fe640 10 monclient: dump: 2024-03-29T11:38:58.232 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:38:58.232 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:38:58.233 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:58.233 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:58.233 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:58.233 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:58.233 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:38:58.233 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:38:58.233 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:38:58.233 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:58.233 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.229+0000 7f4582e1e640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:58.233 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.229+0000 7f4582e1e640 4 set_mon_vals no callback set 2024-03-29T11:38:58.234 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.230+0000 7f4582e1e640 10 monclient: shutdown 2024-03-29T11:38:58.234 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.230+0000 7f4582e1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:58.234 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.230+0000 7f4582e1e640 1 -- 172.21.15.72:0/2188428050 >> v1:172.21.15.72:6789/0 conn(0x7f457c152980 legacy=0x7f457c152e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:58.234 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.230+0000 7f4582e1e640 1 -- 172.21.15.72:0/2188428050 shutdown_connections 2024-03-29T11:38:58.234 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.230+0000 7f4582e1e640 1 -- 172.21.15.72:0/2188428050 >> 172.21.15.72:0/2188428050 conn(0x7f457c0b35e0 msgr2=0x7f457c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:58.234 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.231+0000 7f4582e1e640 1 -- 172.21.15.72:0/2188428050 shutdown_connections 2024-03-29T11:38:58.234 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.233+0000 7f4582e1e640 1 -- 172.21.15.72:0/2188428050 wait complete. 2024-03-29T11:38:58.235 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.234+0000 7f4582e1e640 10 monclient: build_initial_monmap 2024-03-29T11:38:58.235 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.234+0000 7f4582e1e640 10 monclient: monmap: 2024-03-29T11:38:58.235 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:38:58.235 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:58.235 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:38:58.235 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:38:58.235 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:58.235 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:58.235 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:38:58.235 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:38:58.235 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:38:58.235 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:58.236 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.234+0000 7f4582e1e640 1 librados: starting msgr at 2024-03-29T11:38:58.236 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.234+0000 7f4582e1e640 1 librados: starting objecter 2024-03-29T11:38:58.236 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.234+0000 7f4582e1e640 1 Processor -- start 2024-03-29T11:38:58.236 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.235+0000 7f4582e1e640 1 -- start start 2024-03-29T11:38:58.236 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.235+0000 7f4582e1e640 1 librados: setting wanted keys 2024-03-29T11:38:58.237 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.235+0000 7f4582e1e640 1 librados: calling monclient init 2024-03-29T11:38:58.237 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.235+0000 7f4582e1e640 10 monclient: init 2024-03-29T11:38:58.237 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.235+0000 7f4582e1e640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:58.237 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.235+0000 7f4582e1e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:38:58.239 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.238+0000 7f4582e1e640 10 monclient(hunting): picked mon.noname-c con 0x7f457c0af6c0 addr v1:172.21.15.72:6790/0 2024-03-29T11:38:58.239 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.238+0000 7f4582e1e640 10 monclient(hunting): picked mon.noname-a con 0x7f457c14f6b0 addr v1:172.21.15.72:6789/0 2024-03-29T11:38:58.239 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.238+0000 7f4582e1e640 10 monclient(hunting): picked mon.noname-b con 0x7f457c152980 addr v1:172.21.15.89:6789/0 2024-03-29T11:38:58.239 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.238+0000 7f4582e1e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f457c071450 con 0x7f457c14f6b0 2024-03-29T11:38:58.239 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.238+0000 7f4582e1e640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f457c06fcf0 con 0x7f457c152980 2024-03-29T11:38:58.239 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.238+0000 7f4582e1e640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f457c1eff00 con 0x7f457c0af6c0 2024-03-29T11:38:58.239 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.238+0000 7f4582e1e640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:58.239 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.238+0000 7f4582e1e640 10 monclient(hunting): authenticate will time out at 1097.526196s 2024-03-29T11:38:58.241 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.241+0000 7f4580bba640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f457c0af6c0 0x7f457c1512f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:46047/0 (socket says 172.21.15.72:46047) 2024-03-29T11:38:58.242 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.241+0000 7f4580bba640 1 -- 172.21.15.72:0/42084274 learned_addr learned my addr 172.21.15.72:0/42084274 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:38:58.242 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.241+0000 7f45797fa640 10 client.?.objecter ms_handle_connect 0x7f457c0af6c0 2024-03-29T11:38:58.242 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.241+0000 7f45797fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:58.242 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.241+0000 7f45797fa640 10 client.?.objecter ms_handle_connect 0x7f457c14f6b0 2024-03-29T11:38:58.242 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.241+0000 7f45797fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:58.243 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.241+0000 7f45797fa640 1 -- 172.21.15.72:0/42084274 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1820220083 0 0) 0x7f457c1eff00 con 0x7f457c0af6c0 2024-03-29T11:38:58.243 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.241+0000 7f45797fa640 10 monclient(hunting): _negotiate 2024-03-29T11:38:58.243 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.241+0000 7f45797fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:58.243 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.241+0000 7f45797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:58.243 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.241+0000 7f45797fa640 10 monclient(hunting): my global_id is 4370 2024-03-29T11:38:58.243 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.241+0000 7f45797fa640 1 -- 172.21.15.72:0/42084274 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4568004eb0 con 0x7f457c0af6c0 2024-03-29T11:38:58.243 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.241+0000 7f45797fa640 1 -- 172.21.15.72:0/42084274 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4234936460 0 0) 0x7f457c071450 con 0x7f457c14f6b0 2024-03-29T11:38:58.243 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.241+0000 7f45797fa640 10 monclient(hunting): _negotiate 2024-03-29T11:38:58.243 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.241+0000 7f45797fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:58.243 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.241+0000 7f45797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:58.243 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.241+0000 7f45797fa640 10 monclient(hunting): my global_id is 4368 2024-03-29T11:38:58.243 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.241+0000 7f45797fa640 1 -- 172.21.15.72:0/42084274 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f457c1eff00 con 0x7f457c14f6b0 2024-03-29T11:38:58.243 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.241+0000 7f45797fa640 10 client.?.objecter ms_handle_connect 0x7f457c152980 2024-03-29T11:38:58.243 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.241+0000 7f45797fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:58.244 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f45797fa640 1 -- 172.21.15.72:0/42084274 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1070736779 0 0) 0x7f457c06fcf0 con 0x7f457c152980 2024-03-29T11:38:58.244 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f45797fa640 10 monclient(hunting): _negotiate 2024-03-29T11:38:58.244 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f45797fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:58.244 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f45797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:58.244 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f45797fa640 10 monclient(hunting): my global_id is 4354 2024-03-29T11:38:58.244 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f45797fa640 1 -- 172.21.15.72:0/42084274 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f457c071450 con 0x7f457c152980 2024-03-29T11:38:58.244 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f45797fa640 1 -- 172.21.15.72:0/42084274 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4276661353 0 0) 0x7f457c1eff00 con 0x7f457c14f6b0 2024-03-29T11:38:58.244 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f45797fa640 1 -- 172.21.15.72:0/42084274 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f457c06fcf0 con 0x7f457c14f6b0 2024-03-29T11:38:58.244 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f45797fa640 1 -- 172.21.15.72:0/42084274 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f456c001bb0 con 0x7f457c14f6b0 2024-03-29T11:38:58.244 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f45797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:58.245 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f45797fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:38:58.245 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f45797fa640 10 monclient(hunting): dump: 2024-03-29T11:38:58.245 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:38:58.245 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:38:58.245 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:58.245 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:58.245 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:58.245 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:58.245 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:38:58.245 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:38:58.245 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:38:58.245 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:58.245 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f45797fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:58.245 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f45797fa640 1 -- 172.21.15.72:0/42084274 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3397978242 0 0) 0x7f457c071450 con 0x7f457c152980 2024-03-29T11:38:58.245 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f45797fa640 1 -- 172.21.15.72:0/42084274 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f457c1eff00 con 0x7f457c152980 2024-03-29T11:38:58.246 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f45797fa640 1 -- 172.21.15.72:0/42084274 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f45700019d0 con 0x7f457c152980 2024-03-29T11:38:58.246 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f45797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:58.246 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f45797fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:58.246 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f45797fa640 10 monclient(hunting): dump: 2024-03-29T11:38:58.246 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:38:58.246 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:38:58.246 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:58.246 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:58.246 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:58.246 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:58.246 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:38:58.246 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:38:58.246 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:38:58.246 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:58.250 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f45797fa640 1 -- 172.21.15.72:0/42084274 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 803286987 0 0) 0x7f4568004eb0 con 0x7f457c0af6c0 2024-03-29T11:38:58.250 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f45797fa640 1 -- 172.21.15.72:0/42084274 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f457c071450 con 0x7f457c0af6c0 2024-03-29T11:38:58.250 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f45797fa640 1 -- 172.21.15.72:0/42084274 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f4574001990 con 0x7f457c0af6c0 2024-03-29T11:38:58.250 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f45797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:58.250 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f45797fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:58.250 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f45797fa640 10 monclient(hunting): dump: 2024-03-29T11:38:58.250 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:38:58.250 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:38:58.250 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:58.250 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:58.250 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:58.250 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:58.251 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:38:58.251 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:38:58.251 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:38:58.251 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:58.251 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f45797fa640 1 -- 172.21.15.72:0/42084274 <== mon.0 v1:172.21.15.72:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1956330386 0 0) 0x7f457c06fcf0 con 0x7f457c14f6b0 2024-03-29T11:38:58.251 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f45797fa640 1 -- 172.21.15.72:0/42084274 >> v1:172.21.15.72:6790/0 conn(0x7f457c0af6c0 legacy=0x7f457c1512f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:58.251 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f45797fa640 1 -- 172.21.15.72:0/42084274 >> v1:172.21.15.89:6789/0 conn(0x7f457c152980 legacy=0x7f457c071b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:58.251 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f45797fa640 10 monclient: _finish_hunting 0 2024-03-29T11:38:58.251 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f45797fa640 1 monclient: found mon.a 2024-03-29T11:38:58.251 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f45797fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:58.251 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f45797fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:38:58.251 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f45797fa640 1 -- 172.21.15.72:0/42084274 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f457c1f0090 con 0x7f457c14f6b0 2024-03-29T11:38:58.251 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f45797fa640 10 monclient: _finish_auth 0 2024-03-29T11:38:58.251 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f45797fa640 10 monclient: _check_auth_tickets 2024-03-29T11:38:58.251 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f45797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:58.252 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f4582e1e640 5 monclient: authenticate success, global_id 4368 2024-03-29T11:38:58.252 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f4582e1e640 10 monclient: _renew_subs 2024-03-29T11:38:58.252 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f4582e1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:38:58.252 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f4582e1e640 1 -- 172.21.15.72:0/42084274 --> v1:172.21.15.72:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f457c070210 con 0x7f457c14f6b0 2024-03-29T11:38:58.252 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f4582e1e640 10 client.4368.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:58.252 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f4582e1e640 10 monclient: _renew_subs 2024-03-29T11:38:58.252 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f4582e1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:38:58.252 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f4582e1e640 1 -- 172.21.15.72:0/42084274 --> v1:172.21.15.72:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f457c06f9b0 con 0x7f457c14f6b0 2024-03-29T11:38:58.252 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.242+0000 7f4582e1e640 1 librados: init done 2024-03-29T11:38:58.252 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.244+0000 7f4562ffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:58.252 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.244+0000 7f4562ffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:58.252 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.244+0000 7f4562ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:38:58.252 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.244+0000 7f4562ffd640 1 -- 172.21.15.72:0/42084274 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f455c005080 con 0x7f457c14f6b0 2024-03-29T11:38:58.252 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.248+0000 7f45797fa640 1 -- 172.21.15.72:0/42084274 <== mon.0 v1:172.21.15.72:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f456c0020c0 con 0x7f457c14f6b0 2024-03-29T11:38:58.252 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.248+0000 7f45797fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:58.252 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.248+0000 7f45797fa640 1 -- 172.21.15.72:0/42084274 <== mon.0 v1:172.21.15.72:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f456c003b10 con 0x7f457c14f6b0 2024-03-29T11:38:58.253 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.248+0000 7f45797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:58.253 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.248+0000 7f45797fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:58.253 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.248+0000 7f45797fa640 10 monclient: dump: 2024-03-29T11:38:58.253 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:38:58.253 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:38:58.253 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:58.253 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:38:58.253 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:58.253 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:38:58.253 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:38:58.253 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:38:58.253 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:38:58.253 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:38:58.253 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.248+0000 7f457a7fc640 4 set_mon_vals no callback set 2024-03-29T11:38:58.253 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.249+0000 7f45797fa640 1 -- 172.21.15.72:0/42084274 <== mon.0 v1:172.21.15.72:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f456c019ae0 con 0x7f457c14f6b0 2024-03-29T11:38:58.253 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.249+0000 7f4580bba640 10 monclient: get_auth_request con 0x7f456806e070 auth_method 0 2024-03-29T11:38:58.254 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.249+0000 7f45797fa640 1 -- 172.21.15.72:0/42084274 <== mon.0 v1:172.21.15.72:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f456c083a60 con 0x7f457c14f6b0 2024-03-29T11:38:58.254 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.249+0000 7f45797fa640 10 client.4368.objecter ms_dispatch 0x7f457c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:58.255 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.249+0000 7f45797fa640 3 client.4368.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:58.255 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.249+0000 7f45797fa640 3 client.4368.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:58.255 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.250+0000 7f45797fa640 20 client.4368.objecter dump_active .. 0 homeless 2024-03-29T11:38:58.255 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.250+0000 7f45797fa640 10 client.4368.objecter ms_handle_connect 0x7f456806e070 2024-03-29T11:38:58.256 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.255+0000 7f45797fa640 1 -- 172.21.15.72:0/42084274 <== mon.0 v1:172.21.15.72:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f456c04f340 con 0x7f457c14f6b0 2024-03-29T11:38:58.256 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.255+0000 7f45797fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:58.256 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.255+0000 7f45797fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:58.580 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.578+0000 7f4562ffd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:58.581 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.578+0000 7f4562ffd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:58.581 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.578+0000 7f4562ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:38:58.581 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.578+0000 7f4562ffd640 1 -- 172.21.15.72:0/42084274 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f455c005330 con 0x7f457c14f6b0 2024-03-29T11:38:58.581 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.579+0000 7f45797fa640 1 -- 172.21.15.72:0/42084274 <== mon.0 v1:172.21.15.72:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f456c051270 con 0x7f457c14f6b0 2024-03-29T11:38:58.581 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.579+0000 7f45797fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:58.581 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.579+0000 7f45797fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:58.582 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:38:58.582 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:58.586 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.584+0000 7f4562ffd640 10 librados: watch_flush enter 2024-03-29T11:38:58.586 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.585+0000 7f4562ffd640 10 librados: watch_flush exit 2024-03-29T11:38:58.586 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.585+0000 7f4562ffd640 20 client.4368.objecter shutdown clearing up homeless session... 2024-03-29T11:38:58.586 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.585+0000 7f4562ffd640 10 client.4368.objecter successfully canceled tick 2024-03-29T11:38:58.589 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.587+0000 7f4562ffd640 1 -- 172.21.15.72:0/42084274 >> v1:172.21.15.72:6816/1578866102 conn(0x7f456806e070 legacy=0x7f4568070530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:58.589 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.587+0000 7f4562ffd640 10 monclient: shutdown 2024-03-29T11:38:58.589 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.587+0000 7f4562ffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:58.589 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.587+0000 7f4562ffd640 1 -- 172.21.15.72:0/42084274 >> v1:172.21.15.72:6789/0 conn(0x7f457c14f6b0 legacy=0x7f457c1ef7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:58.589 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.587+0000 7f4562ffd640 1 -- 172.21.15.72:0/42084274 shutdown_connections 2024-03-29T11:38:58.589 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.587+0000 7f4562ffd640 1 -- 172.21.15.72:0/42084274 >> 172.21.15.72:0/42084274 conn(0x7f457c0b35e0 msgr2=0x7f457c15e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:58.589 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.588+0000 7f4562ffd640 1 -- 172.21.15.72:0/42084274 shutdown_connections 2024-03-29T11:38:58.589 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.588+0000 7f4562ffd640 1 -- 172.21.15.72:0/42084274 wait complete. 2024-03-29T11:38:58.589 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:38:58.588+0000 7f4562ffd640 1 librados: shutdown 2024-03-29T11:38:58.605 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:05.412 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:05.567 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.565+0000 7f21189af640 10 monclient: get_monmap_and_config 2024-03-29T11:39:05.567 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.565+0000 7f21189af640 10 monclient: build_initial_monmap 2024-03-29T11:39:05.567 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.565+0000 7f21189af640 10 monclient: monmap: 2024-03-29T11:39:05.567 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:39:05.567 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:39:05.568 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:39:05.566338+0000 2024-03-29T11:39:05.568 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:39:05.566338+0000 2024-03-29T11:39:05.568 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:05.568 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:05.568 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:39:05.568 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:39:05.568 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:39:05.568 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:05.569 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.566+0000 7f21189af640 1 Processor -- start 2024-03-29T11:39:05.569 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.566+0000 7f21189af640 1 -- start start 2024-03-29T11:39:05.569 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.566+0000 7f21189af640 10 monclient: init 2024-03-29T11:39:05.569 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.567+0000 7f21189af640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:05.569 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.567+0000 7f21189af640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:39:05.572 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.567+0000 7f21189af640 10 monclient(hunting): picked mon.noname-a con 0x7f21140a9930 addr v1:172.21.15.72:6789/0 2024-03-29T11:39:05.572 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.567+0000 7f21189af640 10 monclient(hunting): picked mon.noname-b con 0x7f2114149dc0 addr v1:172.21.15.72:6790/0 2024-03-29T11:39:05.572 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.567+0000 7f21189af640 10 monclient(hunting): picked mon.noname-c con 0x7f211414ff40 addr v1:172.21.15.89:6789/0 2024-03-29T11:39:05.572 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.567+0000 7f21189af640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21140aae60 con 0x7f21140a9930 2024-03-29T11:39:05.572 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.567+0000 7f21189af640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21140b4310 con 0x7f211414ff40 2024-03-29T11:39:05.572 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.567+0000 7f21189af640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21140afdd0 con 0x7f2114149dc0 2024-03-29T11:39:05.572 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.567+0000 7f21189af640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:05.572 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.567+0000 7f21189af640 10 monclient(hunting): authenticate will time out at 1104.855746s 2024-03-29T11:39:05.574 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.573+0000 7f211259c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f21140a9930 0x7f2114143a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60543/0 (socket says 172.21.15.72:60543) 2024-03-29T11:39:05.574 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.573+0000 7f211259c640 1 -- 172.21.15.72:0/367240292 learned_addr learned my addr 172.21.15.72:0/367240292 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:39:05.579 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.578+0000 7f211159a640 1 -- 172.21.15.72:0/367240292 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1391389800 0 0) 0x7f21140afdd0 con 0x7f2114149dc0 2024-03-29T11:39:05.580 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.578+0000 7f211159a640 10 monclient(hunting): _negotiate 2024-03-29T11:39:05.580 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.578+0000 7f211159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:05.580 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.578+0000 7f211159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:05.580 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.578+0000 7f211159a640 10 monclient(hunting): my global_id is 4373 2024-03-29T11:39:05.580 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.578+0000 7f211159a640 1 -- 172.21.15.72:0/367240292 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20fc003520 con 0x7f2114149dc0 2024-03-29T11:39:05.580 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.578+0000 7f211159a640 1 -- 172.21.15.72:0/367240292 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1447019833 0 0) 0x7f21140aae60 con 0x7f21140a9930 2024-03-29T11:39:05.580 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.578+0000 7f211159a640 10 monclient(hunting): _negotiate 2024-03-29T11:39:05.580 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.578+0000 7f211159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:05.580 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.578+0000 7f211159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:05.580 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.578+0000 7f211159a640 10 monclient(hunting): my global_id is 4371 2024-03-29T11:39:05.580 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.578+0000 7f211159a640 1 -- 172.21.15.72:0/367240292 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21140afdd0 con 0x7f21140a9930 2024-03-29T11:39:05.581 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.579+0000 7f211159a640 1 -- 172.21.15.72:0/367240292 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 782199689 0 0) 0x7f21140afdd0 con 0x7f21140a9930 2024-03-29T11:39:05.581 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.579+0000 7f211159a640 1 -- 172.21.15.72:0/367240292 >> v1:172.21.15.72:6790/0 conn(0x7f2114149dc0 legacy=0x7f211414a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:05.581 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.579+0000 7f211159a640 1 -- 172.21.15.72:0/367240292 >> v1:172.21.15.89:6789/0 conn(0x7f211414ff40 legacy=0x7f2114152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:39:05.581 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.579+0000 7f211159a640 10 monclient: _finish_hunting 0 2024-03-29T11:39:05.581 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.579+0000 7f211159a640 1 monclient: found mon.noname-a 2024-03-29T11:39:05.581 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.579+0000 7f211159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:05.581 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.579+0000 7f211159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.72:6789/0 2024-03-29T11:39:05.581 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.579+0000 7f211159a640 1 -- 172.21.15.72:0/367240292 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f211414a950 con 0x7f21140a9930 2024-03-29T11:39:05.581 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.579+0000 7f211159a640 10 monclient: _finish_auth 0 2024-03-29T11:39:05.581 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.579+0000 7f211159a640 10 monclient: _check_auth_tickets 2024-03-29T11:39:05.581 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.579+0000 7f211159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:05.581 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.579+0000 7f211159a640 1 -- 172.21.15.72:0/367240292 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f210c001970 con 0x7f21140a9930 2024-03-29T11:39:05.581 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.579+0000 7f211159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:05.582 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.579+0000 7f211159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:39:05.582 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.579+0000 7f211159a640 10 monclient: dump: 2024-03-29T11:39:05.582 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:39:05.582 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:39:05.582 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:05.582 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:05.582 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:05.582 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:05.582 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:39:05.582 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:39:05.582 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:39:05.582 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:05.582 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.579+0000 7f21189af640 5 monclient: authenticate success, global_id 4371 2024-03-29T11:39:05.582 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.579+0000 7f21189af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:05.582 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.580+0000 7f211159a640 1 -- 172.21.15.72:0/367240292 <== mon.0 v1:172.21.15.72:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f210c001b90 con 0x7f21140a9930 2024-03-29T11:39:05.582 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.580+0000 7f211159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:05.583 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.580+0000 7f211159a640 1 -- 172.21.15.72:0/367240292 <== mon.0 v1:172.21.15.72:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f210c001f20 con 0x7f21140a9930 2024-03-29T11:39:05.583 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.580+0000 7f211159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:05.583 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.580+0000 7f211159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:05.583 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.580+0000 7f211159a640 10 monclient: dump: 2024-03-29T11:39:05.583 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:39:05.583 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:39:05.583 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:05.583 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:05.583 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:05.583 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:05.583 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:39:05.583 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:39:05.583 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:39:05.583 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:05.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.581+0000 7f21189af640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:05.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.581+0000 7f21189af640 4 set_mon_vals no callback set 2024-03-29T11:39:05.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.582+0000 7f21189af640 10 monclient: shutdown 2024-03-29T11:39:05.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.582+0000 7f21189af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:05.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.582+0000 7f21189af640 1 -- 172.21.15.72:0/367240292 >> v1:172.21.15.72:6789/0 conn(0x7f21140a9930 legacy=0x7f2114143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:05.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.582+0000 7f21189af640 1 -- 172.21.15.72:0/367240292 shutdown_connections 2024-03-29T11:39:05.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.582+0000 7f21189af640 1 -- 172.21.15.72:0/367240292 >> 172.21.15.72:0/367240292 conn(0x7f21140b35e0 msgr2=0x7f21140b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:05.585 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.582+0000 7f21189af640 1 -- 172.21.15.72:0/367240292 shutdown_connections 2024-03-29T11:39:05.586 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.582+0000 7f21189af640 1 -- 172.21.15.72:0/367240292 wait complete. 2024-03-29T11:39:05.586 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.583+0000 7f21189af640 10 monclient: build_initial_monmap 2024-03-29T11:39:05.586 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.583+0000 7f21189af640 10 monclient: monmap: 2024-03-29T11:39:05.586 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:39:05.586 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:05.586 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:39:05.586 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:39:05.586 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:05.586 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:05.586 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:39:05.586 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:39:05.586 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:39:05.586 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:05.586 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.584+0000 7f21189af640 1 librados: starting msgr at 2024-03-29T11:39:05.586 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.584+0000 7f21189af640 1 librados: starting objecter 2024-03-29T11:39:05.586 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.584+0000 7f21189af640 1 Processor -- start 2024-03-29T11:39:05.591 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.584+0000 7f21189af640 1 -- start start 2024-03-29T11:39:05.591 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.584+0000 7f21189af640 1 librados: setting wanted keys 2024-03-29T11:39:05.591 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.584+0000 7f21189af640 1 librados: calling monclient init 2024-03-29T11:39:05.591 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.584+0000 7f21189af640 10 monclient: init 2024-03-29T11:39:05.591 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.585+0000 7f21189af640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:05.591 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.585+0000 7f21189af640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:39:05.591 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.585+0000 7f21189af640 10 monclient(hunting): picked mon.noname-b con 0x7f21140a9930 addr v1:172.21.15.89:6789/0 2024-03-29T11:39:05.591 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.585+0000 7f21189af640 10 monclient(hunting): picked mon.noname-a con 0x7f2114149dc0 addr v1:172.21.15.72:6789/0 2024-03-29T11:39:05.591 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.585+0000 7f21189af640 10 monclient(hunting): picked mon.noname-c con 0x7f211414ff40 addr v1:172.21.15.72:6790/0 2024-03-29T11:39:05.591 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.585+0000 7f21189af640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f211413b120 con 0x7f2114149dc0 2024-03-29T11:39:05.591 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.585+0000 7f21189af640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f211413b400 con 0x7f21140a9930 2024-03-29T11:39:05.592 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.585+0000 7f21189af640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2114148b40 con 0x7f211414ff40 2024-03-29T11:39:05.592 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.585+0000 7f21189af640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:05.592 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.585+0000 7f21189af640 10 monclient(hunting): authenticate will time out at 1104.873672s 2024-03-29T11:39:05.592 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.589+0000 7f211259c640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f21140a9930 0x7f21141e3660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:47557/0 (socket says 172.21.15.72:47557) 2024-03-29T11:39:05.592 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.589+0000 7f211259c640 1 -- 172.21.15.72:0/808607110 learned_addr learned my addr 172.21.15.72:0/808607110 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:39:05.595 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.590+0000 7f2102ffd640 10 client.?.objecter ms_handle_connect 0x7f21140a9930 2024-03-29T11:39:05.595 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.590+0000 7f2102ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:05.595 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.590+0000 7f2102ffd640 10 client.?.objecter ms_handle_connect 0x7f211414ff40 2024-03-29T11:39:05.595 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.590+0000 7f2102ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:05.595 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.590+0000 7f2102ffd640 1 -- 172.21.15.72:0/808607110 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1633585007 0 0) 0x7f211413b400 con 0x7f21140a9930 2024-03-29T11:39:05.595 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.590+0000 7f2102ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:05.595 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.590+0000 7f2102ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:05.595 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.590+0000 7f2102ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:05.595 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.590+0000 7f2102ffd640 10 monclient(hunting): my global_id is 4357 2024-03-29T11:39:05.595 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.590+0000 7f2102ffd640 1 -- 172.21.15.72:0/808607110 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20f4004eb0 con 0x7f21140a9930 2024-03-29T11:39:05.595 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.590+0000 7f2102ffd640 1 -- 172.21.15.72:0/808607110 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2034804387 0 0) 0x7f2114148b40 con 0x7f211414ff40 2024-03-29T11:39:05.596 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.590+0000 7f2102ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:05.596 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.590+0000 7f2102ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:05.596 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.590+0000 7f2102ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:05.596 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.590+0000 7f2102ffd640 10 monclient(hunting): my global_id is 4376 2024-03-29T11:39:05.596 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.590+0000 7f2102ffd640 1 -- 172.21.15.72:0/808607110 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f211413b400 con 0x7f211414ff40 2024-03-29T11:39:05.596 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.591+0000 7f2102ffd640 1 -- 172.21.15.72:0/808607110 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1508444493 0 0) 0x7f211413b400 con 0x7f211414ff40 2024-03-29T11:39:05.596 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.591+0000 7f2102ffd640 1 -- 172.21.15.72:0/808607110 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2114148b40 con 0x7f211414ff40 2024-03-29T11:39:05.596 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.591+0000 7f2102ffd640 1 -- 172.21.15.72:0/808607110 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f2104001640 con 0x7f211414ff40 2024-03-29T11:39:05.596 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.591+0000 7f2102ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:05.596 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.591+0000 7f2102ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:05.596 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.591+0000 7f2102ffd640 10 monclient(hunting): dump: 2024-03-29T11:39:05.596 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:39:05.596 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:39:05.596 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:05.596 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:05.597 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:05.597 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:05.597 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:39:05.597 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:39:05.597 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:39:05.597 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:05.597 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.591+0000 7f2102ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:05.597 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.591+0000 7f2102ffd640 1 -- 172.21.15.72:0/808607110 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3446447100 0 0) 0x7f20f4004eb0 con 0x7f21140a9930 2024-03-29T11:39:05.597 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.591+0000 7f2102ffd640 1 -- 172.21.15.72:0/808607110 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f211413b400 con 0x7f21140a9930 2024-03-29T11:39:05.597 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.591+0000 7f2102ffd640 1 -- 172.21.15.72:0/808607110 <== mon.2 v1:172.21.15.72:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3132364611 0 0) 0x7f2114148b40 con 0x7f211414ff40 2024-03-29T11:39:05.597 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.591+0000 7f2102ffd640 1 -- 172.21.15.72:0/808607110 >> v1:172.21.15.89:6789/0 conn(0x7f21140a9930 legacy=0x7f21141e3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:05.597 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.591+0000 7f2102ffd640 1 -- 172.21.15.72:0/808607110 >> v1:172.21.15.72:6789/0 conn(0x7f2114149dc0 legacy=0x7f21141e7ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:39:05.599 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.594+0000 7f2102ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:39:05.599 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.594+0000 7f2102ffd640 1 monclient: found mon.c 2024-03-29T11:39:05.599 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.594+0000 7f2102ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:05.599 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.594+0000 7f2102ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:39:05.599 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.594+0000 7f2102ffd640 1 -- 172.21.15.72:0/808607110 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2114148cd0 con 0x7f211414ff40 2024-03-29T11:39:05.599 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.594+0000 7f2102ffd640 10 monclient: _finish_auth 0 2024-03-29T11:39:05.599 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.594+0000 7f2102ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:39:05.600 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.594+0000 7f2102ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:05.600 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.595+0000 7f21189af640 5 monclient: authenticate success, global_id 4376 2024-03-29T11:39:05.600 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.595+0000 7f21189af640 10 monclient: _renew_subs 2024-03-29T11:39:05.600 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.595+0000 7f21189af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:39:05.600 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.595+0000 7f21189af640 1 -- 172.21.15.72:0/808607110 --> v1:172.21.15.72:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21140ac490 con 0x7f211414ff40 2024-03-29T11:39:05.600 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.595+0000 7f21189af640 10 client.4376.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:05.600 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.595+0000 7f21189af640 10 monclient: _renew_subs 2024-03-29T11:39:05.600 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.595+0000 7f21189af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:39:05.600 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.595+0000 7f21189af640 1 -- 172.21.15.72:0/808607110 --> v1:172.21.15.72:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f21141495e0 con 0x7f211414ff40 2024-03-29T11:39:05.600 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.595+0000 7f21189af640 1 librados: init done 2024-03-29T11:39:05.600 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.596+0000 7f2102ffd640 1 -- 172.21.15.72:0/808607110 <== mon.2 v1:172.21.15.72:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2104001bd0 con 0x7f211414ff40 2024-03-29T11:39:05.600 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.596+0000 7f2102ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:05.600 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.596+0000 7f2103fff640 4 set_mon_vals no callback set 2024-03-29T11:39:05.600 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.597+0000 7f2102ffd640 1 -- 172.21.15.72:0/808607110 <== mon.2 v1:172.21.15.72:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f2104003790 con 0x7f211414ff40 2024-03-29T11:39:05.600 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.597+0000 7f2102ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:05.601 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.597+0000 7f2102ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:05.601 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.597+0000 7f2102ffd640 10 monclient: dump: 2024-03-29T11:39:05.601 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:39:05.601 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:39:05.601 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:05.601 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:05.601 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:05.601 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:05.601 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:39:05.601 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:39:05.601 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:39:05.601 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:05.602 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.598+0000 7f21189af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:05.602 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.601+0000 7f2102ffd640 1 -- 172.21.15.72:0/808607110 <== mon.2 v1:172.21.15.72:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f21040038f0 con 0x7f211414ff40 2024-03-29T11:39:05.603 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.601+0000 7f21189af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:05.603 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.601+0000 7f21189af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:39:05.603 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.601+0000 7f21189af640 1 -- 172.21.15.72:0/808607110 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20f8005080 con 0x7f211414ff40 2024-03-29T11:39:05.603 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.601+0000 7f2102ffd640 1 -- 172.21.15.72:0/808607110 <== mon.2 v1:172.21.15.72:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f2104083530 con 0x7f211414ff40 2024-03-29T11:39:05.603 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.601+0000 7f2102ffd640 10 client.4376.objecter ms_dispatch 0x7f2114000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:05.603 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.601+0000 7f2102ffd640 3 client.4376.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:05.603 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.601+0000 7f2102ffd640 3 client.4376.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:05.603 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.602+0000 7f2102ffd640 20 client.4376.objecter dump_active .. 0 homeless 2024-03-29T11:39:05.605 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.605+0000 7f211259c640 10 monclient: get_auth_request con 0x7f20f406de90 auth_method 0 2024-03-29T11:39:05.606 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.605+0000 7f2102ffd640 10 client.4376.objecter ms_handle_connect 0x7f20f406de90 2024-03-29T11:39:05.613 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.612+0000 7f2102ffd640 1 -- 172.21.15.72:0/808607110 <== mon.2 v1:172.21.15.72:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f210404ee10 con 0x7f211414ff40 2024-03-29T11:39:05.613 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.612+0000 7f2102ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:05.613 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.612+0000 7f2102ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:05.940 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.937+0000 7f21189af640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:05.940 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.937+0000 7f21189af640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:05.940 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.937+0000 7f21189af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:39:05.940 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.937+0000 7f21189af640 1 -- 172.21.15.72:0/808607110 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20f8005330 con 0x7f211414ff40 2024-03-29T11:39:05.941 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.940+0000 7f2102ffd640 1 -- 172.21.15.72:0/808607110 <== mon.2 v1:172.21.15.72:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f2104001d40 con 0x7f211414ff40 2024-03-29T11:39:05.941 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.940+0000 7f2102ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:05.941 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.940+0000 7f2102ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:05.944 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:39:05.944 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:05.948 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.947+0000 7f21189af640 10 librados: watch_flush enter 2024-03-29T11:39:05.949 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.947+0000 7f21189af640 10 librados: watch_flush exit 2024-03-29T11:39:05.949 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.947+0000 7f21189af640 20 client.4376.objecter shutdown clearing up homeless session... 2024-03-29T11:39:05.949 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.947+0000 7f21189af640 10 client.4376.objecter successfully canceled tick 2024-03-29T11:39:05.949 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.947+0000 7f21189af640 1 -- 172.21.15.72:0/808607110 >> v1:172.21.15.72:6816/1578866102 conn(0x7f20f406de90 legacy=0x7f20f4070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:05.949 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.947+0000 7f21189af640 10 monclient: shutdown 2024-03-29T11:39:05.949 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.947+0000 7f21189af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:05.949 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.947+0000 7f21189af640 1 -- 172.21.15.72:0/808607110 >> v1:172.21.15.72:6790/0 conn(0x7f211414ff40 legacy=0x7f21141e81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:05.949 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.947+0000 7f21189af640 1 -- 172.21.15.72:0/808607110 shutdown_connections 2024-03-29T11:39:05.949 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.947+0000 7f21189af640 1 -- 172.21.15.72:0/808607110 >> 172.21.15.72:0/808607110 conn(0x7f21140b35e0 msgr2=0x7f21140b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:05.949 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.948+0000 7f21189af640 1 -- 172.21.15.72:0/808607110 shutdown_connections 2024-03-29T11:39:05.949 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.948+0000 7f21189af640 1 -- 172.21.15.72:0/808607110 wait complete. 2024-03-29T11:39:05.949 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:05.948+0000 7f21189af640 1 librados: shutdown 2024-03-29T11:39:05.966 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:12.773 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:12.941 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.939+0000 7fbd27fff640 10 monclient: get_monmap_and_config 2024-03-29T11:39:12.941 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.939+0000 7fbd27fff640 10 monclient: build_initial_monmap 2024-03-29T11:39:12.941 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.939+0000 7fbd27fff640 10 monclient: monmap: 2024-03-29T11:39:12.941 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:39:12.941 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:39:12.941 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:39:12.940713+0000 2024-03-29T11:39:12.941 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:39:12.940713+0000 2024-03-29T11:39:12.941 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:12.941 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:12.942 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:39:12.942 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:39:12.942 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:39:12.942 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:12.942 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.940+0000 7fbd27fff640 1 Processor -- start 2024-03-29T11:39:12.942 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.940+0000 7fbd27fff640 1 -- start start 2024-03-29T11:39:12.943 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.940+0000 7fbd27fff640 10 monclient: init 2024-03-29T11:39:12.944 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.942+0000 7fbd27fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:12.944 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.942+0000 7fbd27fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:39:12.944 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.942+0000 7fbd27fff640 10 monclient(hunting): picked mon.noname-c con 0x7fbd20006800 addr v1:172.21.15.89:6789/0 2024-03-29T11:39:12.944 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.942+0000 7fbd27fff640 10 monclient(hunting): picked mon.noname-b con 0x7fbd20006be0 addr v1:172.21.15.72:6790/0 2024-03-29T11:39:12.944 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.942+0000 7fbd27fff640 10 monclient(hunting): picked mon.noname-a con 0x7fbd20031b60 addr v1:172.21.15.72:6789/0 2024-03-29T11:39:12.944 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.942+0000 7fbd27fff640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd20029370 con 0x7fbd20031b60 2024-03-29T11:39:12.944 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.942+0000 7fbd27fff640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd200295e0 con 0x7fbd20006800 2024-03-29T11:39:12.944 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.942+0000 7fbd27fff640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd20031360 con 0x7fbd20006be0 2024-03-29T11:39:12.944 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.942+0000 7fbd27fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:12.944 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.942+0000 7fbd27fff640 10 monclient(hunting): authenticate will time out at 1112.230519s 2024-03-29T11:39:12.944 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.942+0000 7fbd277fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fbd20031b60 0x7fbd200d30c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:43242/0 (socket says 172.21.15.72:43242) 2024-03-29T11:39:12.944 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.942+0000 7fbd277fe640 1 -- 172.21.15.72:0/4091033237 learned_addr learned my addr 172.21.15.72:0/4091033237 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:39:12.945 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.942+0000 7fbd26ffd640 1 --1- 172.21.15.72:0/4091033237 >> v1:172.21.15.89:6789/0 conn(0x7fbd20006800 0x7fbd200cad20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:55880/0 (socket says 172.21.15.72:55880) 2024-03-29T11:39:12.945 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.943+0000 7fbd25ffb640 1 -- 172.21.15.72:0/4091033237 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1009499858 0 0) 0x7fbd200295e0 con 0x7fbd20006800 2024-03-29T11:39:12.945 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.943+0000 7fbd25ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:39:12.945 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.943+0000 7fbd25ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:12.945 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.943+0000 7fbd25ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:12.945 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.943+0000 7fbd25ffb640 10 monclient(hunting): my global_id is 4360 2024-03-29T11:39:12.945 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.943+0000 7fbd25ffb640 1 -- 172.21.15.72:0/4091033237 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd10003520 con 0x7fbd20006800 2024-03-29T11:39:12.947 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.943+0000 7fbd25ffb640 1 -- 172.21.15.72:0/4091033237 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4247539364 0 0) 0x7fbd20029370 con 0x7fbd20031b60 2024-03-29T11:39:12.947 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.943+0000 7fbd25ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:39:12.947 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.943+0000 7fbd25ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:12.947 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.943+0000 7fbd25ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:12.947 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.943+0000 7fbd25ffb640 10 monclient(hunting): my global_id is 4374 2024-03-29T11:39:12.947 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.943+0000 7fbd25ffb640 1 -- 172.21.15.72:0/4091033237 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd200295e0 con 0x7fbd20031b60 2024-03-29T11:39:12.947 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.944+0000 7fbd25ffb640 1 -- 172.21.15.72:0/4091033237 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3691674857 0 0) 0x7fbd10003520 con 0x7fbd20006800 2024-03-29T11:39:12.947 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.944+0000 7fbd25ffb640 1 -- 172.21.15.72:0/4091033237 >> v1:172.21.15.72:6790/0 conn(0x7fbd20006be0 legacy=0x7fbd200c4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:12.947 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.944+0000 7fbd25ffb640 1 -- 172.21.15.72:0/4091033237 >> v1:172.21.15.72:6789/0 conn(0x7fbd20031b60 legacy=0x7fbd200d30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:12.947 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.944+0000 7fbd25ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:39:12.947 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.944+0000 7fbd25ffb640 1 monclient: found mon.noname-c 2024-03-29T11:39:12.947 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.944+0000 7fbd25ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:12.947 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.944+0000 7fbd25ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.89:6789/0 2024-03-29T11:39:12.948 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.944+0000 7fbd25ffb640 1 -- 172.21.15.72:0/4091033237 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd20031640 con 0x7fbd20006800 2024-03-29T11:39:12.948 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.944+0000 7fbd25ffb640 10 monclient: _finish_auth 0 2024-03-29T11:39:12.948 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.944+0000 7fbd25ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:39:12.948 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.944+0000 7fbd25ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:12.948 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.944+0000 7fbd27fff640 5 monclient: authenticate success, global_id 4360 2024-03-29T11:39:12.948 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.944+0000 7fbd25ffb640 1 -- 172.21.15.72:0/4091033237 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fbd2804f730 con 0x7fbd20006800 2024-03-29T11:39:12.948 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.944+0000 7fbd27fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:12.948 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.944+0000 7fbd25ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:12.948 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.944+0000 7fbd25ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:12.948 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.944+0000 7fbd25ffb640 10 monclient: dump: 2024-03-29T11:39:12.948 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:39:12.948 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:39:12.948 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:12.948 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:12.948 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:12.948 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:12.948 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:39:12.948 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:39:12.949 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:39:12.949 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:12.949 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.944+0000 7fbd27fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:12.949 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.944+0000 7fbd25ffb640 1 -- 172.21.15.72:0/4091033237 <== mon.1 v1:172.21.15.89:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd280008d0 con 0x7fbd20006800 2024-03-29T11:39:12.949 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.944+0000 7fbd25ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:12.949 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.944+0000 7fbd25ffb640 1 -- 172.21.15.72:0/4091033237 <== mon.1 v1:172.21.15.89:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fbd280758b0 con 0x7fbd20006800 2024-03-29T11:39:12.949 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.944+0000 7fbd27fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:12.949 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.944+0000 7fbd27fff640 4 set_mon_vals no callback set 2024-03-29T11:39:12.949 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.944+0000 7fbd25ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:12.949 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.944+0000 7fbd25ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:39:12.949 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.944+0000 7fbd25ffb640 10 monclient: dump: 2024-03-29T11:39:12.949 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:39:12.949 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:39:12.949 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:12.949 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:12.950 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:12.950 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:12.950 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:39:12.950 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:39:12.950 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:39:12.950 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:12.950 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.945+0000 7fbd27fff640 10 monclient: shutdown 2024-03-29T11:39:12.950 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.945+0000 7fbd27fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:12.950 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.945+0000 7fbd27fff640 1 -- 172.21.15.72:0/4091033237 >> v1:172.21.15.89:6789/0 conn(0x7fbd20006800 legacy=0x7fbd200cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:12.950 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.945+0000 7fbd27fff640 1 -- 172.21.15.72:0/4091033237 shutdown_connections 2024-03-29T11:39:12.950 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.945+0000 7fbd27fff640 1 -- 172.21.15.72:0/4091033237 >> 172.21.15.72:0/4091033237 conn(0x7fbd200058f0 msgr2=0x7fbd20005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:12.950 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.945+0000 7fbd27fff640 1 -- 172.21.15.72:0/4091033237 shutdown_connections 2024-03-29T11:39:12.950 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.946+0000 7fbd27fff640 1 -- 172.21.15.72:0/4091033237 wait complete. 2024-03-29T11:39:12.953 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.947+0000 7fbd27fff640 10 monclient: build_initial_monmap 2024-03-29T11:39:12.953 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.947+0000 7fbd27fff640 10 monclient: monmap: 2024-03-29T11:39:12.953 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:39:12.953 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:12.953 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:39:12.953 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:39:12.953 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:12.954 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:12.954 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:39:12.954 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:39:12.954 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:39:12.954 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:12.954 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.947+0000 7fbd27fff640 1 librados: starting msgr at 2024-03-29T11:39:12.954 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.947+0000 7fbd27fff640 1 librados: starting objecter 2024-03-29T11:39:12.954 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.947+0000 7fbd27fff640 1 Processor -- start 2024-03-29T11:39:12.954 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.947+0000 7fbd27fff640 1 -- start start 2024-03-29T11:39:12.954 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.947+0000 7fbd27fff640 1 librados: setting wanted keys 2024-03-29T11:39:12.954 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.947+0000 7fbd27fff640 1 librados: calling monclient init 2024-03-29T11:39:12.954 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.947+0000 7fbd27fff640 10 monclient: init 2024-03-29T11:39:12.954 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.948+0000 7fbd27fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:12.954 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.948+0000 7fbd27fff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:39:12.954 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.948+0000 7fbd27fff640 10 monclient(hunting): picked mon.noname-b con 0x7fbd20006800 addr v1:172.21.15.89:6789/0 2024-03-29T11:39:12.955 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.948+0000 7fbd27fff640 10 monclient(hunting): picked mon.noname-a con 0x7fbd20031b60 addr v1:172.21.15.72:6789/0 2024-03-29T11:39:12.955 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.948+0000 7fbd27fff640 10 monclient(hunting): picked mon.noname-c con 0x7fbd200c0600 addr v1:172.21.15.72:6790/0 2024-03-29T11:39:12.955 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.948+0000 7fbd27fff640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd2002f5b0 con 0x7fbd20031b60 2024-03-29T11:39:12.955 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.948+0000 7fbd27fff640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd2002eb60 con 0x7fbd20006800 2024-03-29T11:39:12.955 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.948+0000 7fbd27fff640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd2002edd0 con 0x7fbd200c0600 2024-03-29T11:39:12.955 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.948+0000 7fbd27fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:12.955 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.948+0000 7fbd27fff640 10 monclient(hunting): authenticate will time out at 1112.236946s 2024-03-29T11:39:12.955 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.949+0000 7fbd267fc640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fbd20031b60 0x7fbd2002fa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:43464/0 (socket says 172.21.15.72:43464) 2024-03-29T11:39:12.955 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.949+0000 7fbd267fc640 1 -- 172.21.15.72:0/1760710895 learned_addr learned my addr 172.21.15.72:0/1760710895 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:39:12.955 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.949+0000 7fbd0f7fe640 10 client.?.objecter ms_handle_connect 0x7fbd20031b60 2024-03-29T11:39:12.955 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.949+0000 7fbd0f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:12.955 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.949+0000 7fbd0f7fe640 10 client.?.objecter ms_handle_connect 0x7fbd20006800 2024-03-29T11:39:12.955 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.949+0000 7fbd0f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:12.955 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.949+0000 7fbd0f7fe640 10 client.?.objecter ms_handle_connect 0x7fbd200c0600 2024-03-29T11:39:12.955 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.949+0000 7fbd0f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:12.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.949+0000 7fbd0f7fe640 1 -- 172.21.15.72:0/1760710895 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1690532412 0 0) 0x7fbd2002eb60 con 0x7fbd20006800 2024-03-29T11:39:12.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.950+0000 7fbd0f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:39:12.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.950+0000 7fbd0f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:12.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.950+0000 7fbd0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:12.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.950+0000 7fbd0f7fe640 10 monclient(hunting): my global_id is 4363 2024-03-29T11:39:12.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.950+0000 7fbd0f7fe640 1 -- 172.21.15.72:0/1760710895 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd14004eb0 con 0x7fbd20006800 2024-03-29T11:39:12.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.950+0000 7fbd0f7fe640 1 -- 172.21.15.72:0/1760710895 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 794537704 0 0) 0x7fbd2002f5b0 con 0x7fbd20031b60 2024-03-29T11:39:12.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.950+0000 7fbd0f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:39:12.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.950+0000 7fbd0f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:12.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.950+0000 7fbd0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:12.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.950+0000 7fbd0f7fe640 10 monclient(hunting): my global_id is 4377 2024-03-29T11:39:12.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.950+0000 7fbd0f7fe640 1 -- 172.21.15.72:0/1760710895 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd2002eb60 con 0x7fbd20031b60 2024-03-29T11:39:12.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.950+0000 7fbd0f7fe640 1 -- 172.21.15.72:0/1760710895 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 27796942 0 0) 0x7fbd2002edd0 con 0x7fbd200c0600 2024-03-29T11:39:12.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.950+0000 7fbd0f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:39:12.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.950+0000 7fbd0f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:12.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.950+0000 7fbd0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:12.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.950+0000 7fbd0f7fe640 10 monclient(hunting): my global_id is 4382 2024-03-29T11:39:12.957 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.950+0000 7fbd0f7fe640 1 -- 172.21.15.72:0/1760710895 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd2002f5b0 con 0x7fbd200c0600 2024-03-29T11:39:12.957 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.950+0000 7fbd0f7fe640 1 -- 172.21.15.72:0/1760710895 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 785287988 0 0) 0x7fbd14004eb0 con 0x7fbd20006800 2024-03-29T11:39:12.957 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.950+0000 7fbd0f7fe640 1 -- 172.21.15.72:0/1760710895 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd2002edd0 con 0x7fbd20006800 2024-03-29T11:39:12.957 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.950+0000 7fbd0f7fe640 1 -- 172.21.15.72:0/1760710895 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fbd2804f400 con 0x7fbd20006800 2024-03-29T11:39:12.957 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.950+0000 7fbd0f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:12.957 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.950+0000 7fbd0f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:12.957 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.950+0000 7fbd0f7fe640 10 monclient(hunting): dump: 2024-03-29T11:39:12.957 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:39:12.957 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:39:12.957 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:12.957 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:12.957 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:12.957 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:12.957 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:39:12.957 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:39:12.957 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:39:12.957 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:12.958 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.950+0000 7fbd0f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:12.958 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.951+0000 7fbd0f7fe640 1 -- 172.21.15.72:0/1760710895 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3178887490 0 0) 0x7fbd2002eb60 con 0x7fbd20031b60 2024-03-29T11:39:12.958 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.951+0000 7fbd0f7fe640 1 -- 172.21.15.72:0/1760710895 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd14004eb0 con 0x7fbd20031b60 2024-03-29T11:39:12.958 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.951+0000 7fbd0f7fe640 1 -- 172.21.15.72:0/1760710895 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fbd180019d0 con 0x7fbd20031b60 2024-03-29T11:39:12.958 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.951+0000 7fbd0f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:12.958 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.951+0000 7fbd0f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:12.958 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.951+0000 7fbd0f7fe640 10 monclient(hunting): dump: 2024-03-29T11:39:12.958 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:39:12.958 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:39:12.958 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:12.958 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:12.958 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:12.958 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:12.958 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:39:12.958 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:39:12.958 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:39:12.959 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:12.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.951+0000 7fbd0f7fe640 1 -- 172.21.15.72:0/1760710895 <== mon.1 v1:172.21.15.89:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1425754910 0 0) 0x7fbd2002edd0 con 0x7fbd20006800 2024-03-29T11:39:12.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.951+0000 7fbd0f7fe640 1 -- 172.21.15.72:0/1760710895 >> v1:172.21.15.72:6790/0 conn(0x7fbd200c0600 legacy=0x7fbd20030150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:12.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.951+0000 7fbd0f7fe640 1 -- 172.21.15.72:0/1760710895 >> v1:172.21.15.72:6789/0 conn(0x7fbd20031b60 legacy=0x7fbd2002fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:12.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.951+0000 7fbd0f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:39:12.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.951+0000 7fbd0f7fe640 1 monclient: found mon.b 2024-03-29T11:39:12.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.951+0000 7fbd0f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:12.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.951+0000 7fbd0f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:39:12.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.951+0000 7fbd0f7fe640 1 -- 172.21.15.72:0/1760710895 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd2002f1f0 con 0x7fbd20006800 2024-03-29T11:39:12.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.951+0000 7fbd0f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:39:12.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.951+0000 7fbd0f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:39:12.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.951+0000 7fbd0f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:12.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.951+0000 7fbd27fff640 5 monclient: authenticate success, global_id 4363 2024-03-29T11:39:12.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.951+0000 7fbd27fff640 10 monclient: _renew_subs 2024-03-29T11:39:12.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.951+0000 7fbd27fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:39:12.960 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.951+0000 7fbd27fff640 1 -- 172.21.15.72:0/1760710895 --> v1:172.21.15.89:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd2002c9e0 con 0x7fbd20006800 2024-03-29T11:39:12.960 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.951+0000 7fbd27fff640 10 client.4363.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:12.960 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.951+0000 7fbd27fff640 10 monclient: _renew_subs 2024-03-29T11:39:12.960 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.951+0000 7fbd27fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:39:12.960 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.951+0000 7fbd27fff640 1 -- 172.21.15.72:0/1760710895 --> v1:172.21.15.89:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd2002d790 con 0x7fbd20006800 2024-03-29T11:39:12.960 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.951+0000 7fbd27fff640 1 librados: init done 2024-03-29T11:39:12.960 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.952+0000 7fbd0f7fe640 1 -- 172.21.15.72:0/1760710895 <== mon.1 v1:172.21.15.89:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd2804fdd0 con 0x7fbd20006800 2024-03-29T11:39:12.960 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.952+0000 7fbd0f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:12.960 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.952+0000 7fbd0f7fe640 1 -- 172.21.15.72:0/1760710895 <== mon.1 v1:172.21.15.89:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fbd28067bf0 con 0x7fbd20006800 2024-03-29T11:39:12.960 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.952+0000 7fbd0f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:12.960 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.952+0000 7fbd0f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:39:12.960 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.952+0000 7fbd0f7fe640 10 monclient: dump: 2024-03-29T11:39:12.960 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:39:12.960 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:39:12.960 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:12.960 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:12.961 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:12.961 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:12.961 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:39:12.961 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:39:12.961 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:39:12.961 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:12.961 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.952+0000 7fbd257fa640 4 set_mon_vals no callback set 2024-03-29T11:39:12.961 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.952+0000 7fbd27fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:12.961 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.953+0000 7fbd27fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:12.961 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.953+0000 7fbd27fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:39:12.962 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.953+0000 7fbd27fff640 1 -- 172.21.15.72:0/1760710895 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd08005080 con 0x7fbd20006800 2024-03-29T11:39:12.962 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.954+0000 7fbd0f7fe640 1 -- 172.21.15.72:0/1760710895 <== mon.1 v1:172.21.15.89:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7fbd28067d50 con 0x7fbd20006800 2024-03-29T11:39:12.962 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.954+0000 7fbd0f7fe640 1 -- 172.21.15.72:0/1760710895 <== mon.1 v1:172.21.15.89:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7fbd280ff490 con 0x7fbd20006800 2024-03-29T11:39:12.962 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.954+0000 7fbd0f7fe640 10 client.4363.objecter ms_dispatch 0x7fbd28000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:12.962 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.954+0000 7fbd0f7fe640 3 client.4363.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:12.962 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.954+0000 7fbd0f7fe640 3 client.4363.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:12.962 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.954+0000 7fbd0f7fe640 20 client.4363.objecter dump_active .. 0 homeless 2024-03-29T11:39:12.962 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.955+0000 7fbd267fc640 10 monclient: get_auth_request con 0x7fbd1406dd30 auth_method 0 2024-03-29T11:39:12.962 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.955+0000 7fbd0f7fe640 10 client.4363.objecter ms_handle_connect 0x7fbd1406dd30 2024-03-29T11:39:12.962 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.961+0000 7fbd0f7fe640 1 -- 172.21.15.72:0/1760710895 <== mon.1 v1:172.21.15.89:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbd280cad70 con 0x7fbd20006800 2024-03-29T11:39:12.963 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.961+0000 7fbd0f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:12.963 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:12.961+0000 7fbd0f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:13.323 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:13.322+0000 7fbd27fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:13.323 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:13.322+0000 7fbd27fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:13.323 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:13.322+0000 7fbd27fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:39:13.323 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:13.322+0000 7fbd27fff640 1 -- 172.21.15.72:0/1760710895 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbd08005330 con 0x7fbd20006800 2024-03-29T11:39:13.323 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:13.323+0000 7fbd0f7fe640 1 -- 172.21.15.72:0/1760710895 <== mon.1 v1:172.21.15.89:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7fbd280ccca0 con 0x7fbd20006800 2024-03-29T11:39:13.323 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:13.323+0000 7fbd0f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:13.324 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:13.323+0000 7fbd0f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:13.324 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:39:13.324 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:13.328 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:13.326+0000 7fbd27fff640 10 librados: watch_flush enter 2024-03-29T11:39:13.328 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:13.326+0000 7fbd27fff640 10 librados: watch_flush exit 2024-03-29T11:39:13.328 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:13.326+0000 7fbd27fff640 20 client.4363.objecter shutdown clearing up homeless session... 2024-03-29T11:39:13.328 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:13.326+0000 7fbd27fff640 10 client.4363.objecter successfully canceled tick 2024-03-29T11:39:13.328 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:13.327+0000 7fbd27fff640 1 -- 172.21.15.72:0/1760710895 >> v1:172.21.15.72:6816/1578866102 conn(0x7fbd1406dd30 legacy=0x7fbd140701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:13.328 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:13.327+0000 7fbd27fff640 10 monclient: shutdown 2024-03-29T11:39:13.328 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:13.327+0000 7fbd27fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:13.328 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:13.327+0000 7fbd27fff640 1 -- 172.21.15.72:0/1760710895 >> v1:172.21.15.89:6789/0 conn(0x7fbd20006800 legacy=0x7fbd200bfef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:13.328 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:13.327+0000 7fbd27fff640 1 -- 172.21.15.72:0/1760710895 shutdown_connections 2024-03-29T11:39:13.329 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:13.327+0000 7fbd27fff640 1 -- 172.21.15.72:0/1760710895 >> 172.21.15.72:0/1760710895 conn(0x7fbd200058f0 msgr2=0x7fbd200c4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:13.329 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:13.327+0000 7fbd27fff640 1 -- 172.21.15.72:0/1760710895 shutdown_connections 2024-03-29T11:39:13.329 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:13.327+0000 7fbd27fff640 1 -- 172.21.15.72:0/1760710895 wait complete. 2024-03-29T11:39:13.329 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:13.327+0000 7fbd27fff640 1 librados: shutdown 2024-03-29T11:39:13.343 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:20.150 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:20.302 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.300+0000 7f7c2eff6640 10 monclient: get_monmap_and_config 2024-03-29T11:39:20.302 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.300+0000 7f7c2eff6640 10 monclient: build_initial_monmap 2024-03-29T11:39:20.302 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.300+0000 7f7c2eff6640 10 monclient: monmap: 2024-03-29T11:39:20.302 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:39:20.302 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:39:20.302 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:39:20.301752+0000 2024-03-29T11:39:20.302 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:39:20.301752+0000 2024-03-29T11:39:20.302 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:20.302 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:20.302 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:39:20.302 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:39:20.303 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:39:20.303 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:20.305 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.302+0000 7f7c2eff6640 1 Processor -- start 2024-03-29T11:39:20.305 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.302+0000 7f7c2eff6640 1 -- start start 2024-03-29T11:39:20.305 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.302+0000 7f7c2eff6640 10 monclient: init 2024-03-29T11:39:20.306 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.303+0000 7f7c2eff6640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:20.306 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.303+0000 7f7c2eff6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:39:20.306 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.303+0000 7f7c2eff6640 10 monclient(hunting): picked mon.noname-c con 0x7f7c281422b0 addr v1:172.21.15.89:6789/0 2024-03-29T11:39:20.306 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.303+0000 7f7c2eff6640 10 monclient(hunting): picked mon.noname-a con 0x7f7c28142690 addr v1:172.21.15.72:6789/0 2024-03-29T11:39:20.306 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.303+0000 7f7c2eff6640 10 monclient(hunting): picked mon.noname-b con 0x7f7c280b17d0 addr v1:172.21.15.72:6790/0 2024-03-29T11:39:20.306 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.303+0000 7f7c2eff6640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c28142c80 con 0x7f7c28142690 2024-03-29T11:39:20.306 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.303+0000 7f7c2eff6640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c280aae40 con 0x7f7c281422b0 2024-03-29T11:39:20.306 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.303+0000 7f7c2eff6640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c2813fbd0 con 0x7f7c280b17d0 2024-03-29T11:39:20.306 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.303+0000 7f7c2eff6640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:20.306 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.303+0000 7f7c2eff6640 10 monclient(hunting): authenticate will time out at 1119.591963s 2024-03-29T11:39:20.311 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.310+0000 7f7c2cd92640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f7c281422b0 0x7f7c2813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:48609/0 (socket says 172.21.15.72:48609) 2024-03-29T11:39:20.311 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.310+0000 7f7c2cd92640 1 -- 172.21.15.72:0/3177529873 learned_addr learned my addr 172.21.15.72:0/3177529873 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:39:20.311 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.310+0000 7f7c277fe640 1 -- 172.21.15.72:0/3177529873 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 733854685 0 0) 0x7f7c280aae40 con 0x7f7c281422b0 2024-03-29T11:39:20.311 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.310+0000 7f7c277fe640 10 monclient(hunting): _negotiate 2024-03-29T11:39:20.312 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.310+0000 7f7c277fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:20.312 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.310+0000 7f7c277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:20.312 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.310+0000 7f7c277fe640 10 monclient(hunting): my global_id is 4366 2024-03-29T11:39:20.312 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.310+0000 7f7c277fe640 1 -- 172.21.15.72:0/3177529873 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c10003520 con 0x7f7c281422b0 2024-03-29T11:39:20.312 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.311+0000 7f7c277fe640 1 -- 172.21.15.72:0/3177529873 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 645587218 0 0) 0x7f7c10003520 con 0x7f7c281422b0 2024-03-29T11:39:20.312 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.311+0000 7f7c277fe640 1 -- 172.21.15.72:0/3177529873 >> v1:172.21.15.72:6790/0 conn(0x7f7c280b17d0 legacy=0x7f7c280b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:39:20.312 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.311+0000 7f7c277fe640 1 -- 172.21.15.72:0/3177529873 >> v1:172.21.15.72:6789/0 conn(0x7f7c28142690 legacy=0x7f7c280b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:39:20.312 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.311+0000 7f7c277fe640 10 monclient: _finish_hunting 0 2024-03-29T11:39:20.313 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.311+0000 7f7c277fe640 1 monclient: found mon.noname-c 2024-03-29T11:39:20.313 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.311+0000 7f7c277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:20.313 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.311+0000 7f7c277fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.89:6789/0 2024-03-29T11:39:20.313 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.311+0000 7f7c277fe640 1 -- 172.21.15.72:0/3177529873 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c28151940 con 0x7f7c281422b0 2024-03-29T11:39:20.313 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.311+0000 7f7c277fe640 10 monclient: _finish_auth 0 2024-03-29T11:39:20.313 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.311+0000 7f7c277fe640 10 monclient: _check_auth_tickets 2024-03-29T11:39:20.313 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.311+0000 7f7c277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:20.313 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.311+0000 7f7c277fe640 1 -- 172.21.15.72:0/3177529873 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f7c1c001670 con 0x7f7c281422b0 2024-03-29T11:39:20.313 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.311+0000 7f7c277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:20.313 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.311+0000 7f7c277fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:20.313 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.311+0000 7f7c277fe640 10 monclient: dump: 2024-03-29T11:39:20.313 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:39:20.313 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:39:20.313 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:20.313 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:20.314 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:20.314 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:20.314 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:39:20.314 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:39:20.314 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:39:20.314 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:20.314 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.311+0000 7f7c277fe640 1 -- 172.21.15.72:0/3177529873 <== mon.1 v1:172.21.15.89:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c1c0018d0 con 0x7f7c281422b0 2024-03-29T11:39:20.314 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.311+0000 7f7c277fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:20.314 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.311+0000 7f7c277fe640 1 -- 172.21.15.72:0/3177529873 <== mon.1 v1:172.21.15.89:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f7c1c001670 con 0x7f7c281422b0 2024-03-29T11:39:20.314 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.311+0000 7f7c277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:20.314 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.311+0000 7f7c277fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:39:20.314 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.311+0000 7f7c277fe640 10 monclient: dump: 2024-03-29T11:39:20.314 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:39:20.314 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:39:20.314 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:20.314 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:20.314 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:20.315 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:20.315 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:39:20.315 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:39:20.315 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:39:20.315 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:20.315 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.312+0000 7f7c2eff6640 5 monclient: authenticate success, global_id 4366 2024-03-29T11:39:20.315 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.312+0000 7f7c2eff6640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:20.315 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.312+0000 7f7c2eff6640 4 set_mon_vals no callback set 2024-03-29T11:39:20.315 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.312+0000 7f7c2eff6640 10 monclient: shutdown 2024-03-29T11:39:20.315 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.312+0000 7f7c2eff6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:20.315 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.312+0000 7f7c2eff6640 1 -- 172.21.15.72:0/3177529873 >> v1:172.21.15.89:6789/0 conn(0x7f7c281422b0 legacy=0x7f7c2813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:20.316 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.315+0000 7f7c2eff6640 1 -- 172.21.15.72:0/3177529873 shutdown_connections 2024-03-29T11:39:20.316 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.315+0000 7f7c2eff6640 1 -- 172.21.15.72:0/3177529873 >> 172.21.15.72:0/3177529873 conn(0x7f7c28066a00 msgr2=0x7f7c28066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:20.317 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.316+0000 7f7c2eff6640 1 -- 172.21.15.72:0/3177529873 shutdown_connections 2024-03-29T11:39:20.318 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.317+0000 7f7c2eff6640 1 -- 172.21.15.72:0/3177529873 wait complete. 2024-03-29T11:39:20.319 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.318+0000 7f7c2eff6640 10 monclient: build_initial_monmap 2024-03-29T11:39:20.320 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.318+0000 7f7c2eff6640 10 monclient: monmap: 2024-03-29T11:39:20.320 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:39:20.320 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:20.320 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:39:20.320 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:39:20.320 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:20.320 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:20.320 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:39:20.320 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:39:20.320 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:39:20.320 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:20.320 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.318+0000 7f7c2eff6640 1 librados: starting msgr at 2024-03-29T11:39:20.320 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.318+0000 7f7c2eff6640 1 librados: starting objecter 2024-03-29T11:39:20.320 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.318+0000 7f7c2eff6640 1 Processor -- start 2024-03-29T11:39:20.321 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.318+0000 7f7c2eff6640 1 -- start start 2024-03-29T11:39:20.321 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.318+0000 7f7c2eff6640 1 librados: setting wanted keys 2024-03-29T11:39:20.321 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.318+0000 7f7c2eff6640 1 librados: calling monclient init 2024-03-29T11:39:20.321 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.318+0000 7f7c2eff6640 10 monclient: init 2024-03-29T11:39:20.322 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.319+0000 7f7c2eff6640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:20.322 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.319+0000 7f7c2eff6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:39:20.322 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.319+0000 7f7c2eff6640 10 monclient(hunting): picked mon.noname-c con 0x7f7c280b17d0 addr v1:172.21.15.72:6790/0 2024-03-29T11:39:20.322 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.319+0000 7f7c2eff6640 10 monclient(hunting): picked mon.noname-b con 0x7f7c281422b0 addr v1:172.21.15.89:6789/0 2024-03-29T11:39:20.322 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.320+0000 7f7c2eff6640 10 monclient(hunting): picked mon.noname-a con 0x7f7c28142690 addr v1:172.21.15.72:6789/0 2024-03-29T11:39:20.322 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.320+0000 7f7c2eff6640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c2814d750 con 0x7f7c28142690 2024-03-29T11:39:20.322 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.320+0000 7f7c2eff6640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c280b5c00 con 0x7f7c281422b0 2024-03-29T11:39:20.323 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.320+0000 7f7c2eff6640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c280b5ee0 con 0x7f7c280b17d0 2024-03-29T11:39:20.323 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.320+0000 7f7c2eff6640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:20.323 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.320+0000 7f7c2eff6640 10 monclient(hunting): authenticate will time out at 1119.608236s 2024-03-29T11:39:20.326 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.325+0000 7f7c2cd92640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f7c280b17d0 0x7f7c2814cee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:36351/0 (socket says 172.21.15.72:36351) 2024-03-29T11:39:20.327 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.325+0000 7f7c2cd92640 1 -- 172.21.15.72:0/2444755907 learned_addr learned my addr 172.21.15.72:0/2444755907 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:39:20.327 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.325+0000 7f7c257fa640 10 client.?.objecter ms_handle_connect 0x7f7c280b17d0 2024-03-29T11:39:20.327 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.326+0000 7f7c257fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:20.330 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.327+0000 7f7c257fa640 1 -- 172.21.15.72:0/2444755907 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3906448832 0 0) 0x7f7c280b5ee0 con 0x7f7c280b17d0 2024-03-29T11:39:20.330 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.327+0000 7f7c257fa640 10 monclient(hunting): _negotiate 2024-03-29T11:39:20.331 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.327+0000 7f7c257fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:20.331 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.327+0000 7f7c257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:20.331 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.327+0000 7f7c257fa640 10 monclient(hunting): my global_id is 4385 2024-03-29T11:39:20.331 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.327+0000 7f7c257fa640 1 -- 172.21.15.72:0/2444755907 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c14004eb0 con 0x7f7c280b17d0 2024-03-29T11:39:20.331 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.327+0000 7f7c257fa640 10 client.?.objecter ms_handle_connect 0x7f7c281422b0 2024-03-29T11:39:20.331 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.327+0000 7f7c257fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:20.331 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.328+0000 7f7c257fa640 10 client.?.objecter ms_handle_connect 0x7f7c28142690 2024-03-29T11:39:20.331 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.328+0000 7f7c257fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:20.331 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.328+0000 7f7c257fa640 1 -- 172.21.15.72:0/2444755907 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 778426504 0 0) 0x7f7c280b5c00 con 0x7f7c281422b0 2024-03-29T11:39:20.331 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.328+0000 7f7c257fa640 10 monclient(hunting): _negotiate 2024-03-29T11:39:20.331 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.328+0000 7f7c257fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:20.331 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.328+0000 7f7c257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:20.331 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.328+0000 7f7c257fa640 10 monclient(hunting): my global_id is 4369 2024-03-29T11:39:20.331 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.328+0000 7f7c257fa640 1 -- 172.21.15.72:0/2444755907 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c280b5ee0 con 0x7f7c281422b0 2024-03-29T11:39:20.332 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.329+0000 7f7c257fa640 1 -- 172.21.15.72:0/2444755907 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 253885927 0 0) 0x7f7c2814d750 con 0x7f7c28142690 2024-03-29T11:39:20.332 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.329+0000 7f7c257fa640 10 monclient(hunting): _negotiate 2024-03-29T11:39:20.332 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.329+0000 7f7c257fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:20.332 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.329+0000 7f7c257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:20.332 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.329+0000 7f7c257fa640 10 monclient(hunting): my global_id is 4380 2024-03-29T11:39:20.332 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.329+0000 7f7c257fa640 1 -- 172.21.15.72:0/2444755907 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c280b5c00 con 0x7f7c28142690 2024-03-29T11:39:20.333 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.330+0000 7f7c257fa640 1 -- 172.21.15.72:0/2444755907 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2353638550 0 0) 0x7f7c14004eb0 con 0x7f7c280b17d0 2024-03-29T11:39:20.333 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.330+0000 7f7c257fa640 1 -- 172.21.15.72:0/2444755907 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c2814d750 con 0x7f7c280b17d0 2024-03-29T11:39:20.333 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.330+0000 7f7c257fa640 1 -- 172.21.15.72:0/2444755907 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 526050377 0 0) 0x7f7c280b5ee0 con 0x7f7c281422b0 2024-03-29T11:39:20.333 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.330+0000 7f7c257fa640 1 -- 172.21.15.72:0/2444755907 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c14004eb0 con 0x7f7c281422b0 2024-03-29T11:39:20.333 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.330+0000 7f7c257fa640 1 -- 172.21.15.72:0/2444755907 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f7c1c001670 con 0x7f7c280b17d0 2024-03-29T11:39:20.333 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.330+0000 7f7c257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:20.333 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.330+0000 7f7c257fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:20.333 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.330+0000 7f7c257fa640 10 monclient(hunting): dump: 2024-03-29T11:39:20.333 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:39:20.333 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:39:20.333 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:20.333 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:20.333 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:20.333 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:20.334 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:39:20.334 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:39:20.334 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:39:20.334 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:20.334 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.330+0000 7f7c257fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:20.334 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.330+0000 7f7c257fa640 1 -- 172.21.15.72:0/2444755907 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f7c20001a70 con 0x7f7c281422b0 2024-03-29T11:39:20.334 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.330+0000 7f7c257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:20.334 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.330+0000 7f7c257fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:39:20.334 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.330+0000 7f7c257fa640 10 monclient(hunting): dump: 2024-03-29T11:39:20.334 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:39:20.334 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:39:20.334 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:20.334 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:20.334 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:20.334 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:20.335 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:39:20.335 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:39:20.335 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:39:20.335 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:20.335 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.331+0000 7f7c257fa640 1 -- 172.21.15.72:0/2444755907 <== mon.1 v1:172.21.15.89:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 974948934 0 0) 0x7f7c14004eb0 con 0x7f7c281422b0 2024-03-29T11:39:20.335 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.331+0000 7f7c257fa640 1 -- 172.21.15.72:0/2444755907 >> v1:172.21.15.72:6790/0 conn(0x7f7c280b17d0 legacy=0x7f7c2814cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:20.335 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.331+0000 7f7c257fa640 1 -- 172.21.15.72:0/2444755907 >> v1:172.21.15.72:6789/0 conn(0x7f7c28142690 legacy=0x7f7c280b76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:20.335 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.331+0000 7f7c257fa640 10 monclient: _finish_hunting 0 2024-03-29T11:39:20.335 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.331+0000 7f7c257fa640 1 monclient: found mon.b 2024-03-29T11:39:20.335 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.331+0000 7f7c257fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:20.335 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.331+0000 7f7c257fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:39:20.335 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.331+0000 7f7c257fa640 1 -- 172.21.15.72:0/2444755907 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c2814e2f0 con 0x7f7c281422b0 2024-03-29T11:39:20.335 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.331+0000 7f7c257fa640 10 monclient: _finish_auth 0 2024-03-29T11:39:20.335 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.331+0000 7f7c257fa640 10 monclient: _check_auth_tickets 2024-03-29T11:39:20.335 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.331+0000 7f7c257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:20.335 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.331+0000 7f7c2eff6640 5 monclient: authenticate success, global_id 4369 2024-03-29T11:39:20.336 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.331+0000 7f7c2eff6640 10 monclient: _renew_subs 2024-03-29T11:39:20.336 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.331+0000 7f7c2eff6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:39:20.336 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.331+0000 7f7c2eff6640 1 -- 172.21.15.72:0/2444755907 --> v1:172.21.15.89:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c281de9d0 con 0x7f7c281422b0 2024-03-29T11:39:20.336 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.331+0000 7f7c2eff6640 10 client.4369.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:20.336 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.331+0000 7f7c2eff6640 10 monclient: _renew_subs 2024-03-29T11:39:20.336 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.331+0000 7f7c2eff6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:39:20.336 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.331+0000 7f7c2eff6640 1 -- 172.21.15.72:0/2444755907 --> v1:172.21.15.89:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c280b7fc0 con 0x7f7c281422b0 2024-03-29T11:39:20.336 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.331+0000 7f7c2eff6640 1 librados: init done 2024-03-29T11:39:20.336 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.331+0000 7f7c257fa640 1 -- 172.21.15.72:0/2444755907 <== mon.1 v1:172.21.15.89:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c20001f60 con 0x7f7c281422b0 2024-03-29T11:39:20.336 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.332+0000 7f7c257fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:20.336 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.332+0000 7f7c257fa640 1 -- 172.21.15.72:0/2444755907 <== mon.1 v1:172.21.15.89:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f7c20003b10 con 0x7f7c281422b0 2024-03-29T11:39:20.336 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.332+0000 7f7c257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:20.336 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.332+0000 7f7c257fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:39:20.336 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.332+0000 7f7c257fa640 10 monclient: dump: 2024-03-29T11:39:20.336 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:39:20.336 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:39:20.337 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:20.337 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:20.337 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:20.337 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:20.337 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:39:20.337 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:39:20.337 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:39:20.337 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:20.338 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.333+0000 7f7c267fc640 4 set_mon_vals no callback set 2024-03-29T11:39:20.338 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.333+0000 7f7c2eff6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:20.338 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.334+0000 7f7c2eff6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:20.338 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.334+0000 7f7c2eff6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:39:20.338 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.334+0000 7f7c2eff6640 1 -- 172.21.15.72:0/2444755907 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c08005080 con 0x7f7c281422b0 2024-03-29T11:39:20.338 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.335+0000 7f7c257fa640 1 -- 172.21.15.72:0/2444755907 <== mon.1 v1:172.21.15.89:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f7c20019ae0 con 0x7f7c281422b0 2024-03-29T11:39:20.338 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.336+0000 7f7c2cd92640 10 monclient: get_auth_request con 0x7f7c1406dfe0 auth_method 0 2024-03-29T11:39:20.338 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.336+0000 7f7c257fa640 10 client.4369.objecter ms_handle_connect 0x7f7c1406dfe0 2024-03-29T11:39:20.338 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.336+0000 7f7c257fa640 1 -- 172.21.15.72:0/2444755907 <== mon.1 v1:172.21.15.89:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f7c20083a60 con 0x7f7c281422b0 2024-03-29T11:39:20.339 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.337+0000 7f7c257fa640 10 client.4369.objecter ms_dispatch 0x7f7c28000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:20.339 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.337+0000 7f7c257fa640 3 client.4369.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:20.339 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.337+0000 7f7c257fa640 3 client.4369.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:20.339 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.337+0000 7f7c257fa640 20 client.4369.objecter dump_active .. 0 homeless 2024-03-29T11:39:20.343 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.342+0000 7f7c257fa640 1 -- 172.21.15.72:0/2444755907 <== mon.1 v1:172.21.15.89:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7c2004f340 con 0x7f7c281422b0 2024-03-29T11:39:20.343 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.342+0000 7f7c257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:20.343 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.342+0000 7f7c257fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:20.693 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.691+0000 7f7c2eff6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:20.693 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.691+0000 7f7c2eff6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:20.693 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.691+0000 7f7c2eff6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:39:20.693 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.691+0000 7f7c2eff6640 1 -- 172.21.15.72:0/2444755907 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c08005330 con 0x7f7c281422b0 2024-03-29T11:39:20.693 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.692+0000 7f7c257fa640 1 -- 172.21.15.72:0/2444755907 <== mon.1 v1:172.21.15.89:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f7c20051270 con 0x7f7c281422b0 2024-03-29T11:39:20.694 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.693+0000 7f7c257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:20.694 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.693+0000 7f7c257fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:20.694 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:39:20.694 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:20.700 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.697+0000 7f7c2eff6640 10 librados: watch_flush enter 2024-03-29T11:39:20.701 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.699+0000 7f7c2eff6640 10 librados: watch_flush exit 2024-03-29T11:39:20.701 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.699+0000 7f7c2eff6640 20 client.4369.objecter shutdown clearing up homeless session... 2024-03-29T11:39:20.701 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.699+0000 7f7c2eff6640 10 client.4369.objecter successfully canceled tick 2024-03-29T11:39:20.701 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.699+0000 7f7c2eff6640 1 -- 172.21.15.72:0/2444755907 >> v1:172.21.15.72:6816/1578866102 conn(0x7f7c1406dfe0 legacy=0x7f7c140704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:20.701 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.699+0000 7f7c2eff6640 10 monclient: shutdown 2024-03-29T11:39:20.701 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.699+0000 7f7c2eff6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:20.701 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.699+0000 7f7c2eff6640 1 -- 172.21.15.72:0/2444755907 >> v1:172.21.15.89:6789/0 conn(0x7f7c281422b0 legacy=0x7f7c2814dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:20.701 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.699+0000 7f7c2eff6640 1 -- 172.21.15.72:0/2444755907 shutdown_connections 2024-03-29T11:39:20.701 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.699+0000 7f7c2eff6640 1 -- 172.21.15.72:0/2444755907 >> 172.21.15.72:0/2444755907 conn(0x7f7c28066a00 msgr2=0x7f7c28066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:20.701 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.699+0000 7f7c2eff6640 1 -- 172.21.15.72:0/2444755907 shutdown_connections 2024-03-29T11:39:20.701 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.699+0000 7f7c2eff6640 1 -- 172.21.15.72:0/2444755907 wait complete. 2024-03-29T11:39:20.701 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:20.699+0000 7f7c2eff6640 1 librados: shutdown 2024-03-29T11:39:20.717 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:27.524 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:27.661 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.660+0000 7f1b043b5640 10 monclient: get_monmap_and_config 2024-03-29T11:39:27.662 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.660+0000 7f1b043b5640 10 monclient: build_initial_monmap 2024-03-29T11:39:27.662 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.660+0000 7f1b043b5640 10 monclient: monmap: 2024-03-29T11:39:27.662 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:39:27.662 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:39:27.662 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:39:27.661625+0000 2024-03-29T11:39:27.662 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:39:27.661625+0000 2024-03-29T11:39:27.662 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:27.662 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:27.662 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:39:27.662 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:39:27.662 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:39:27.662 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:27.663 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.661+0000 7f1b043b5640 1 Processor -- start 2024-03-29T11:39:27.663 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.661+0000 7f1b043b5640 1 -- start start 2024-03-29T11:39:27.663 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.661+0000 7f1b043b5640 10 monclient: init 2024-03-29T11:39:27.666 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.662+0000 7f1b043b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:27.666 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.662+0000 7f1b043b5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:39:27.666 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.662+0000 7f1b043b5640 10 monclient(hunting): picked mon.noname-a con 0x7f1afc1457f0 addr v1:172.21.15.72:6789/0 2024-03-29T11:39:27.666 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.663+0000 7f1b043b5640 10 monclient(hunting): picked mon.noname-c con 0x7f1afc13f650 addr v1:172.21.15.89:6789/0 2024-03-29T11:39:27.667 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.663+0000 7f1b043b5640 10 monclient(hunting): picked mon.noname-b con 0x7f1afc0a9d70 addr v1:172.21.15.72:6790/0 2024-03-29T11:39:27.667 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.663+0000 7f1b02151640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f1afc1457f0 0x7f1afc13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:35798/0 (socket says 172.21.15.72:35798) 2024-03-29T11:39:27.667 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.663+0000 7f1b043b5640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1afc0aaef0 con 0x7f1afc1457f0 2024-03-29T11:39:27.667 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.663+0000 7f1b02151640 1 -- 172.21.15.72:0/4255474752 learned_addr learned my addr 172.21.15.72:0/4255474752 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:39:27.667 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.663+0000 7f1b043b5640 1 -- 172.21.15.72:0/4255474752 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1afc145bd0 con 0x7f1afc13f650 2024-03-29T11:39:27.667 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.663+0000 7f1b043b5640 1 -- 172.21.15.72:0/4255474752 --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1afc145eb0 con 0x7f1afc0a9d70 2024-03-29T11:39:27.667 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.663+0000 7f1b043b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:27.667 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.663+0000 7f1b043b5640 10 monclient(hunting): authenticate will time out at 1126.951411s 2024-03-29T11:39:27.667 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.664+0000 7f1b0114f640 1 -- 172.21.15.72:0/4255474752 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4205664494 0 0) 0x7f1afc0aaef0 con 0x7f1afc1457f0 2024-03-29T11:39:27.667 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.664+0000 7f1b0114f640 10 monclient(hunting): _negotiate 2024-03-29T11:39:27.667 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.664+0000 7f1b0114f640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:27.667 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.664+0000 7f1b0114f640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:27.667 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.664+0000 7f1b0114f640 10 monclient(hunting): my global_id is 4383 2024-03-29T11:39:27.668 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.664+0000 7f1b0114f640 1 -- 172.21.15.72:0/4255474752 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ae8003520 con 0x7f1afc1457f0 2024-03-29T11:39:27.668 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.664+0000 7f1b0114f640 1 -- 172.21.15.72:0/4255474752 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 802478376 0 0) 0x7f1afc145eb0 con 0x7f1afc0a9d70 2024-03-29T11:39:27.668 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.664+0000 7f1b0114f640 10 monclient(hunting): _negotiate 2024-03-29T11:39:27.668 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.664+0000 7f1b0114f640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:27.668 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.664+0000 7f1b0114f640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:27.668 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.664+0000 7f1b0114f640 10 monclient(hunting): my global_id is 4388 2024-03-29T11:39:27.668 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.664+0000 7f1b0114f640 1 -- 172.21.15.72:0/4255474752 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1afc0aaef0 con 0x7f1afc0a9d70 2024-03-29T11:39:27.668 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.664+0000 7f1b0114f640 1 -- 172.21.15.72:0/4255474752 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3226986410 0 0) 0x7f1ae8003520 con 0x7f1afc1457f0 2024-03-29T11:39:27.668 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.664+0000 7f1b0114f640 1 -- 172.21.15.72:0/4255474752 >> v1:172.21.15.72:6790/0 conn(0x7f1afc0a9d70 legacy=0x7f1afc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:27.668 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.664+0000 7f1b0114f640 1 -- 172.21.15.72:0/4255474752 >> v1:172.21.15.89:6789/0 conn(0x7f1afc13f650 legacy=0x7f1afc0b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:39:27.668 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.665+0000 7f1b0114f640 10 monclient: _finish_hunting 0 2024-03-29T11:39:27.668 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.665+0000 7f1b0114f640 1 monclient: found mon.noname-a 2024-03-29T11:39:27.668 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.665+0000 7f1b0114f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:27.669 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.665+0000 7f1b0114f640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.72:6789/0 2024-03-29T11:39:27.669 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.665+0000 7f1b0114f640 1 -- 172.21.15.72:0/4255474752 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1afc146340 con 0x7f1afc1457f0 2024-03-29T11:39:27.669 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.665+0000 7f1b0114f640 10 monclient: _finish_auth 0 2024-03-29T11:39:27.669 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.665+0000 7f1b0114f640 10 monclient: _check_auth_tickets 2024-03-29T11:39:27.669 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.665+0000 7f1b0114f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:27.669 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.665+0000 7f1b0114f640 1 -- 172.21.15.72:0/4255474752 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f1af8001690 con 0x7f1afc1457f0 2024-03-29T11:39:27.669 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.665+0000 7f1b0114f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:27.669 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.665+0000 7f1b0114f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:39:27.669 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.665+0000 7f1b0114f640 10 monclient: dump: 2024-03-29T11:39:27.669 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:39:27.669 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:39:27.669 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:27.669 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:27.670 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:27.670 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:27.670 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:39:27.670 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:39:27.670 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:39:27.670 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:27.670 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.665+0000 7f1b043b5640 5 monclient: authenticate success, global_id 4383 2024-03-29T11:39:27.670 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.665+0000 7f1b043b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:27.675 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.666+0000 7f1b0114f640 1 -- 172.21.15.72:0/4255474752 <== mon.0 v1:172.21.15.72:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1af80017f0 con 0x7f1afc1457f0 2024-03-29T11:39:27.675 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.666+0000 7f1b0114f640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:27.675 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.666+0000 7f1b0114f640 1 -- 172.21.15.72:0/4255474752 <== mon.0 v1:172.21.15.72:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f1af8001c00 con 0x7f1afc1457f0 2024-03-29T11:39:27.676 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.666+0000 7f1b0114f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:27.676 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.666+0000 7f1b0114f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:27.676 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.666+0000 7f1b0114f640 10 monclient: dump: 2024-03-29T11:39:27.676 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:39:27.676 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:39:27.676 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:27.676 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:27.676 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:27.676 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:27.676 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:39:27.676 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:39:27.676 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:39:27.676 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:27.676 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.666+0000 7f1b043b5640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:27.677 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.666+0000 7f1b043b5640 4 set_mon_vals no callback set 2024-03-29T11:39:27.677 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.667+0000 7f1b043b5640 10 monclient: shutdown 2024-03-29T11:39:27.677 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.667+0000 7f1b043b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:27.677 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.667+0000 7f1b043b5640 1 -- 172.21.15.72:0/4255474752 >> v1:172.21.15.72:6789/0 conn(0x7f1afc1457f0 legacy=0x7f1afc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:27.677 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.667+0000 7f1b043b5640 1 -- 172.21.15.72:0/4255474752 shutdown_connections 2024-03-29T11:39:27.677 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.667+0000 7f1b043b5640 1 -- 172.21.15.72:0/4255474752 >> 172.21.15.72:0/4255474752 conn(0x7f1afc066a00 msgr2=0x7f1afc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:27.677 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.667+0000 7f1b043b5640 1 -- 172.21.15.72:0/4255474752 shutdown_connections 2024-03-29T11:39:27.677 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.667+0000 7f1b043b5640 1 -- 172.21.15.72:0/4255474752 wait complete. 2024-03-29T11:39:27.677 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.668+0000 7f1b043b5640 10 monclient: build_initial_monmap 2024-03-29T11:39:27.677 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.668+0000 7f1b043b5640 10 monclient: monmap: 2024-03-29T11:39:27.677 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:39:27.677 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:27.677 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:39:27.677 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:39:27.677 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:27.677 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:27.678 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:39:27.678 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:39:27.678 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:39:27.678 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:27.678 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.669+0000 7f1b043b5640 1 librados: starting msgr at 2024-03-29T11:39:27.678 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.669+0000 7f1b043b5640 1 librados: starting objecter 2024-03-29T11:39:27.678 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.669+0000 7f1b043b5640 1 Processor -- start 2024-03-29T11:39:27.678 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.669+0000 7f1b043b5640 1 -- start start 2024-03-29T11:39:27.678 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.669+0000 7f1b043b5640 1 librados: setting wanted keys 2024-03-29T11:39:27.678 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.669+0000 7f1b043b5640 1 librados: calling monclient init 2024-03-29T11:39:27.678 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.669+0000 7f1b043b5640 10 monclient: init 2024-03-29T11:39:27.678 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.670+0000 7f1b043b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:27.678 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.670+0000 7f1b043b5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:39:27.678 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.670+0000 7f1b043b5640 10 monclient(hunting): picked mon.noname-b con 0x7f1afc0a9d70 addr v1:172.21.15.89:6789/0 2024-03-29T11:39:27.679 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.670+0000 7f1b043b5640 10 monclient(hunting): picked mon.noname-a con 0x7f1afc13f650 addr v1:172.21.15.72:6789/0 2024-03-29T11:39:27.679 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.670+0000 7f1b043b5640 10 monclient(hunting): picked mon.noname-c con 0x7f1afc1457f0 addr v1:172.21.15.72:6790/0 2024-03-29T11:39:27.679 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.670+0000 7f1b043b5640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1afc1dc050 con 0x7f1afc13f650 2024-03-29T11:39:27.679 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.670+0000 7f1b043b5640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1afc06f9a0 con 0x7f1afc0a9d70 2024-03-29T11:39:27.679 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.670+0000 7f1b043b5640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1afc1dc800 con 0x7f1afc1457f0 2024-03-29T11:39:27.679 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.670+0000 7f1b043b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:27.679 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.670+0000 7f1b043b5640 10 monclient(hunting): authenticate will time out at 1126.958752s 2024-03-29T11:39:27.679 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.671+0000 7f1b02952640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f1afc1457f0 0x7f1afc06ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:37376/0 (socket says 172.21.15.72:37376) 2024-03-29T11:39:27.679 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.671+0000 7f1b02952640 1 -- 172.21.15.72:0/3791514821 learned_addr learned my addr 172.21.15.72:0/3791514821 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:39:27.679 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.671+0000 7f1af2ffd640 10 client.?.objecter ms_handle_connect 0x7f1afc1457f0 2024-03-29T11:39:27.679 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.671+0000 7f1af2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:27.679 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.671+0000 7f1af2ffd640 10 client.?.objecter ms_handle_connect 0x7f1afc13f650 2024-03-29T11:39:27.679 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.671+0000 7f1af2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:27.679 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.671+0000 7f1af2ffd640 1 -- 172.21.15.72:0/3791514821 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2321200885 0 0) 0x7f1afc1dc050 con 0x7f1afc13f650 2024-03-29T11:39:27.679 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.671+0000 7f1af2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:27.680 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.671+0000 7f1af2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:27.680 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.671+0000 7f1af2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:27.680 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.671+0000 7f1af2ffd640 10 monclient(hunting): my global_id is 4386 2024-03-29T11:39:27.680 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.671+0000 7f1af2ffd640 1 -- 172.21.15.72:0/3791514821 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ae0004eb0 con 0x7f1afc13f650 2024-03-29T11:39:27.680 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.672+0000 7f1af2ffd640 1 -- 172.21.15.72:0/3791514821 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1451728284 0 0) 0x7f1ae0004eb0 con 0x7f1afc13f650 2024-03-29T11:39:27.680 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.672+0000 7f1af2ffd640 1 -- 172.21.15.72:0/3791514821 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1afc1dc050 con 0x7f1afc13f650 2024-03-29T11:39:27.680 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.672+0000 7f1af2ffd640 1 -- 172.21.15.72:0/3791514821 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f1aec001870 con 0x7f1afc13f650 2024-03-29T11:39:27.680 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.672+0000 7f1af2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:27.680 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.672+0000 7f1af2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:39:27.680 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.672+0000 7f1af2ffd640 10 monclient(hunting): dump: 2024-03-29T11:39:27.680 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:39:27.680 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:39:27.680 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:27.680 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:27.680 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:27.680 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:27.680 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:39:27.680 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:39:27.681 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:39:27.681 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:27.681 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.672+0000 7f1af2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:27.681 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.672+0000 7f1af2ffd640 1 -- 172.21.15.72:0/3791514821 <== mon.0 v1:172.21.15.72:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2099054416 0 0) 0x7f1afc1dc050 con 0x7f1afc13f650 2024-03-29T11:39:27.681 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.672+0000 7f1af2ffd640 1 -- 172.21.15.72:0/3791514821 >> v1:172.21.15.72:6790/0 conn(0x7f1afc1457f0 legacy=0x7f1afc06ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:27.681 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.672+0000 7f1af2ffd640 1 -- 172.21.15.72:0/3791514821 >> v1:172.21.15.89:6789/0 conn(0x7f1afc0a9d70 legacy=0x7f1afc071a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:39:27.681 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.673+0000 7f1af2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:39:27.681 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.673+0000 7f1af2ffd640 1 monclient: found mon.a 2024-03-29T11:39:27.681 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.673+0000 7f1af2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:27.681 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.673+0000 7f1af2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:39:27.681 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.673+0000 7f1af2ffd640 1 -- 172.21.15.72:0/3791514821 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1afc0712c0 con 0x7f1afc13f650 2024-03-29T11:39:27.681 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.673+0000 7f1af2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:39:27.681 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.673+0000 7f1af2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:39:27.681 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.673+0000 7f1af2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:27.681 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.673+0000 7f1b043b5640 5 monclient: authenticate success, global_id 4386 2024-03-29T11:39:27.681 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.673+0000 7f1b043b5640 10 monclient: _renew_subs 2024-03-29T11:39:27.681 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.673+0000 7f1b043b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:39:27.681 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.673+0000 7f1b043b5640 1 -- 172.21.15.72:0/3791514821 --> v1:172.21.15.72:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1afc1dc2e0 con 0x7f1afc13f650 2024-03-29T11:39:27.682 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.673+0000 7f1b043b5640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:27.682 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.673+0000 7f1b043b5640 10 monclient: _renew_subs 2024-03-29T11:39:27.682 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.673+0000 7f1b043b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:39:27.682 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.673+0000 7f1b043b5640 1 -- 172.21.15.72:0/3791514821 --> v1:172.21.15.72:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1afc0b0650 con 0x7f1afc13f650 2024-03-29T11:39:27.682 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.673+0000 7f1b043b5640 1 librados: init done 2024-03-29T11:39:27.683 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.675+0000 7f1af2ffd640 1 -- 172.21.15.72:0/3791514821 <== mon.0 v1:172.21.15.72:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1aec001e00 con 0x7f1afc13f650 2024-03-29T11:39:27.683 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.675+0000 7f1af2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:27.683 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.675+0000 7f1af2ffd640 1 -- 172.21.15.72:0/3791514821 <== mon.0 v1:172.21.15.72:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f1aec003830 con 0x7f1afc13f650 2024-03-29T11:39:27.683 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.675+0000 7f1af2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:27.683 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.675+0000 7f1af2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:27.683 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.675+0000 7f1af2ffd640 10 monclient: dump: 2024-03-29T11:39:27.683 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:39:27.684 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:39:27.684 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:27.684 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:27.684 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:27.684 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:27.684 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:39:27.684 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:39:27.684 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:39:27.684 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:27.684 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.675+0000 7f1b0094e640 4 set_mon_vals no callback set 2024-03-29T11:39:27.684 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.675+0000 7f1b043b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:27.684 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.676+0000 7f1b043b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:27.684 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.676+0000 7f1b043b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:39:27.684 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.676+0000 7f1b043b5640 1 -- 172.21.15.72:0/3791514821 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ae4005080 con 0x7f1afc13f650 2024-03-29T11:39:27.684 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.676+0000 7f1af2ffd640 1 -- 172.21.15.72:0/3791514821 <== mon.0 v1:172.21.15.72:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f1aec0198c0 con 0x7f1afc13f650 2024-03-29T11:39:27.684 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.677+0000 7f1af2ffd640 1 -- 172.21.15.72:0/3791514821 <== mon.0 v1:172.21.15.72:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f1aec083860 con 0x7f1afc13f650 2024-03-29T11:39:27.684 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.677+0000 7f1af2ffd640 10 client.4386.objecter ms_dispatch 0x7f1afc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:27.685 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.677+0000 7f1af2ffd640 3 client.4386.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:27.685 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.677+0000 7f1af2ffd640 3 client.4386.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:27.685 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.677+0000 7f1b02151640 10 monclient: get_auth_request con 0x7f1ae006dc20 auth_method 0 2024-03-29T11:39:27.685 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.677+0000 7f1af2ffd640 20 client.4386.objecter dump_active .. 0 homeless 2024-03-29T11:39:27.685 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.677+0000 7f1af2ffd640 10 client.4386.objecter ms_handle_connect 0x7f1ae006dc20 2024-03-29T11:39:27.686 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.684+0000 7f1af2ffd640 1 -- 172.21.15.72:0/3791514821 <== mon.0 v1:172.21.15.72:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1aec04ef70 con 0x7f1afc13f650 2024-03-29T11:39:27.686 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.684+0000 7f1af2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:27.686 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.684+0000 7f1af2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:27.990 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.988+0000 7f1b043b5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:27.990 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.988+0000 7f1b043b5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:27.990 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.988+0000 7f1b043b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:39:27.990 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.988+0000 7f1b043b5640 1 -- 172.21.15.72:0/3791514821 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ae4005330 con 0x7f1afc13f650 2024-03-29T11:39:27.991 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.989+0000 7f1af2ffd640 1 -- 172.21.15.72:0/3791514821 <== mon.0 v1:172.21.15.72:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f1aec051070 con 0x7f1afc13f650 2024-03-29T11:39:27.991 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.989+0000 7f1af2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:27.991 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.989+0000 7f1af2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:27.991 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:39:27.991 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:27.994 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.992+0000 7f1b043b5640 10 librados: watch_flush enter 2024-03-29T11:39:27.994 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.992+0000 7f1b043b5640 10 librados: watch_flush exit 2024-03-29T11:39:27.995 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.992+0000 7f1b043b5640 20 client.4386.objecter shutdown clearing up homeless session... 2024-03-29T11:39:27.995 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.992+0000 7f1b043b5640 10 client.4386.objecter successfully canceled tick 2024-03-29T11:39:27.995 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.992+0000 7f1b043b5640 1 -- 172.21.15.72:0/3791514821 >> v1:172.21.15.72:6816/1578866102 conn(0x7f1ae006dc20 legacy=0x7f1ae00700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:27.995 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.992+0000 7f1b043b5640 10 monclient: shutdown 2024-03-29T11:39:27.995 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.992+0000 7f1b043b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:27.995 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.992+0000 7f1b043b5640 1 -- 172.21.15.72:0/3791514821 >> v1:172.21.15.72:6789/0 conn(0x7f1afc13f650 legacy=0x7f1afc1db940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:27.995 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.992+0000 7f1b043b5640 1 -- 172.21.15.72:0/3791514821 shutdown_connections 2024-03-29T11:39:27.995 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.992+0000 7f1b043b5640 1 -- 172.21.15.72:0/3791514821 >> 172.21.15.72:0/3791514821 conn(0x7f1afc066a00 msgr2=0x7f1afc0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:27.995 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.993+0000 7f1b043b5640 1 -- 172.21.15.72:0/3791514821 shutdown_connections 2024-03-29T11:39:27.995 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.993+0000 7f1b043b5640 1 -- 172.21.15.72:0/3791514821 wait complete. 2024-03-29T11:39:27.995 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:27.993+0000 7f1b043b5640 1 librados: shutdown 2024-03-29T11:39:28.008 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:34.815 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:34.967 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.965+0000 7fbe2b889640 10 monclient: get_monmap_and_config 2024-03-29T11:39:34.968 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.965+0000 7fbe2b889640 10 monclient: build_initial_monmap 2024-03-29T11:39:34.968 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.965+0000 7fbe2b889640 10 monclient: monmap: 2024-03-29T11:39:34.968 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:39:34.968 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:39:34.968 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:39:34.967252+0000 2024-03-29T11:39:34.968 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:39:34.967252+0000 2024-03-29T11:39:34.968 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:34.968 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:34.968 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:39:34.968 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:39:34.968 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:39:34.968 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:34.972 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.966+0000 7fbe2b889640 1 Processor -- start 2024-03-29T11:39:34.972 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.966+0000 7fbe2b889640 1 -- start start 2024-03-29T11:39:34.972 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.966+0000 7fbe2b889640 10 monclient: init 2024-03-29T11:39:34.972 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.967+0000 7fbe2b889640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:34.972 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.967+0000 7fbe2b889640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:39:34.972 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.967+0000 7fbe2b889640 10 monclient(hunting): picked mon.noname-b con 0x7fbe241457f0 addr v1:172.21.15.72:6790/0 2024-03-29T11:39:34.972 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.967+0000 7fbe2b889640 10 monclient(hunting): picked mon.noname-c con 0x7fbe2413f650 addr v1:172.21.15.89:6789/0 2024-03-29T11:39:34.972 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.967+0000 7fbe2b889640 10 monclient(hunting): picked mon.noname-a con 0x7fbe240a9d70 addr v1:172.21.15.72:6789/0 2024-03-29T11:39:34.973 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.967+0000 7fbe2b889640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe240aaef0 con 0x7fbe240a9d70 2024-03-29T11:39:34.973 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.967+0000 7fbe2b889640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe24145bd0 con 0x7fbe2413f650 2024-03-29T11:39:34.973 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.967+0000 7fbe2b889640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe24145eb0 con 0x7fbe241457f0 2024-03-29T11:39:34.973 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.967+0000 7fbe2b889640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:34.973 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.967+0000 7fbe2b889640 10 monclient(hunting): authenticate will time out at 1134.256459s 2024-03-29T11:39:34.973 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.967+0000 7fbe29e26640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fbe240a9d70 0x7fbe2414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:52296/0 (socket says 172.21.15.72:52296) 2024-03-29T11:39:34.973 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.967+0000 7fbe29e26640 1 -- 172.21.15.72:0/1218831200 learned_addr learned my addr 172.21.15.72:0/1218831200 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:39:34.973 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.967+0000 7fbe29625640 1 --1- 172.21.15.72:0/1218831200 >> v1:172.21.15.72:6790/0 conn(0x7fbe241457f0 0x7fbe2413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:38702/0 (socket says 172.21.15.72:38702) 2024-03-29T11:39:34.973 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.968+0000 7fbe1bfff640 1 -- 172.21.15.72:0/1218831200 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4150961459 0 0) 0x7fbe24145eb0 con 0x7fbe241457f0 2024-03-29T11:39:34.973 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.968+0000 7fbe1bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:34.973 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.968+0000 7fbe1bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:34.973 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.968+0000 7fbe1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:34.973 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.968+0000 7fbe1bfff640 10 monclient(hunting): my global_id is 4391 2024-03-29T11:39:34.973 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.968+0000 7fbe1bfff640 1 -- 172.21.15.72:0/1218831200 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe10003520 con 0x7fbe241457f0 2024-03-29T11:39:34.973 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.968+0000 7fbe1bfff640 1 -- 172.21.15.72:0/1218831200 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1292064625 0 0) 0x7fbe10003520 con 0x7fbe241457f0 2024-03-29T11:39:34.974 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.968+0000 7fbe1bfff640 1 -- 172.21.15.72:0/1218831200 >> v1:172.21.15.89:6789/0 conn(0x7fbe2413f650 legacy=0x7fbe240b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:34.974 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.968+0000 7fbe1bfff640 1 -- 172.21.15.72:0/1218831200 >> v1:172.21.15.72:6789/0 conn(0x7fbe240a9d70 legacy=0x7fbe2414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:34.974 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.969+0000 7fbe1bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:39:34.974 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.969+0000 7fbe1bfff640 1 monclient: found mon.noname-b 2024-03-29T11:39:34.974 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.969+0000 7fbe1bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:34.974 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.969+0000 7fbe1bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.72:6790/0 2024-03-29T11:39:34.974 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.969+0000 7fbe1bfff640 1 -- 172.21.15.72:0/1218831200 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe24146340 con 0x7fbe241457f0 2024-03-29T11:39:34.974 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.969+0000 7fbe1bfff640 10 monclient: _finish_auth 0 2024-03-29T11:39:34.974 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.969+0000 7fbe1bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:39:34.974 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.969+0000 7fbe1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:34.975 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.969+0000 7fbe2b889640 5 monclient: authenticate success, global_id 4391 2024-03-29T11:39:34.975 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.969+0000 7fbe2b889640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:34.975 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.969+0000 7fbe1bfff640 1 -- 172.21.15.72:0/1218831200 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fbe14001690 con 0x7fbe241457f0 2024-03-29T11:39:34.975 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.969+0000 7fbe1bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:34.975 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.969+0000 7fbe1bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:34.975 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.969+0000 7fbe1bfff640 10 monclient: dump: 2024-03-29T11:39:34.975 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:39:34.975 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:39:34.975 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:34.975 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:34.975 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:34.975 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:34.975 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:39:34.975 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:39:34.975 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:39:34.976 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:34.976 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.969+0000 7fbe2b889640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:34.976 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.969+0000 7fbe1bfff640 1 -- 172.21.15.72:0/1218831200 <== mon.2 v1:172.21.15.72:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe140017f0 con 0x7fbe241457f0 2024-03-29T11:39:34.976 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.969+0000 7fbe1bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:34.976 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.969+0000 7fbe2b889640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:34.976 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.969+0000 7fbe2b889640 4 set_mon_vals no callback set 2024-03-29T11:39:34.976 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.970+0000 7fbe2b889640 10 monclient: shutdown 2024-03-29T11:39:34.976 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.970+0000 7fbe2b889640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:34.976 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.970+0000 7fbe2b889640 1 -- 172.21.15.72:0/1218831200 >> v1:172.21.15.72:6790/0 conn(0x7fbe241457f0 legacy=0x7fbe2413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:34.981 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.970+0000 7fbe2b889640 1 -- 172.21.15.72:0/1218831200 shutdown_connections 2024-03-29T11:39:34.981 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.970+0000 7fbe2b889640 1 -- 172.21.15.72:0/1218831200 >> 172.21.15.72:0/1218831200 conn(0x7fbe24066a00 msgr2=0x7fbe24066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:34.981 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.970+0000 7fbe2b889640 1 -- 172.21.15.72:0/1218831200 shutdown_connections 2024-03-29T11:39:34.981 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.970+0000 7fbe2b889640 1 -- 172.21.15.72:0/1218831200 wait complete. 2024-03-29T11:39:34.981 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.971+0000 7fbe2b889640 10 monclient: build_initial_monmap 2024-03-29T11:39:34.981 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.971+0000 7fbe2b889640 10 monclient: monmap: 2024-03-29T11:39:34.981 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:39:34.981 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:34.981 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:39:34.981 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:39:34.981 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:34.981 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:34.982 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:39:34.982 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:39:34.982 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:39:34.982 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:34.982 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.972+0000 7fbe2b889640 1 librados: starting msgr at 2024-03-29T11:39:34.982 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.972+0000 7fbe2b889640 1 librados: starting objecter 2024-03-29T11:39:34.982 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.972+0000 7fbe2b889640 1 Processor -- start 2024-03-29T11:39:34.982 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.972+0000 7fbe2b889640 1 -- start start 2024-03-29T11:39:34.982 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.972+0000 7fbe2b889640 1 librados: setting wanted keys 2024-03-29T11:39:34.982 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.972+0000 7fbe2b889640 1 librados: calling monclient init 2024-03-29T11:39:34.982 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.972+0000 7fbe2b889640 10 monclient: init 2024-03-29T11:39:34.982 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.973+0000 7fbe2b889640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:34.982 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.973+0000 7fbe2b889640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:39:34.982 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.973+0000 7fbe2b889640 10 monclient(hunting): picked mon.noname-a con 0x7fbe240a9d70 addr v1:172.21.15.72:6789/0 2024-03-29T11:39:34.982 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.973+0000 7fbe2b889640 10 monclient(hunting): picked mon.noname-c con 0x7fbe2413f650 addr v1:172.21.15.72:6790/0 2024-03-29T11:39:34.983 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.973+0000 7fbe2b889640 10 monclient(hunting): picked mon.noname-b con 0x7fbe241457f0 addr v1:172.21.15.89:6789/0 2024-03-29T11:39:34.983 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.973+0000 7fbe2b889640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe241ceb20 con 0x7fbe240a9d70 2024-03-29T11:39:34.983 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.973+0000 7fbe2b889640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe241cd1d0 con 0x7fbe241457f0 2024-03-29T11:39:34.983 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.973+0000 7fbe2b889640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe241ccd60 con 0x7fbe2413f650 2024-03-29T11:39:34.983 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.973+0000 7fbe2b889640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:34.983 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.973+0000 7fbe2b889640 10 monclient(hunting): authenticate will time out at 1134.262626s 2024-03-29T11:39:34.983 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.973+0000 7fbe29625640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fbe240a9d70 0x7fbe241cd390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:54170/0 (socket says 172.21.15.72:54170) 2024-03-29T11:39:34.983 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.973+0000 7fbe29625640 1 -- 172.21.15.72:0/3864036851 learned_addr learned my addr 172.21.15.72:0/3864036851 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:39:34.983 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.973+0000 7fbe19ffb640 10 client.?.objecter ms_handle_connect 0x7fbe240a9d70 2024-03-29T11:39:34.983 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.973+0000 7fbe19ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:34.983 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.973+0000 7fbe19ffb640 10 client.?.objecter ms_handle_connect 0x7fbe2413f650 2024-03-29T11:39:34.983 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.973+0000 7fbe19ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:34.983 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.974+0000 7fbe19ffb640 1 -- 172.21.15.72:0/3864036851 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2038588563 0 0) 0x7fbe241ceb20 con 0x7fbe240a9d70 2024-03-29T11:39:34.983 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.974+0000 7fbe19ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:39:34.983 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.974+0000 7fbe19ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:34.983 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.974+0000 7fbe19ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:34.984 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.974+0000 7fbe19ffb640 10 monclient(hunting): my global_id is 4392 2024-03-29T11:39:34.984 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.974+0000 7fbe19ffb640 1 -- 172.21.15.72:0/3864036851 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe08004eb0 con 0x7fbe240a9d70 2024-03-29T11:39:34.984 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.974+0000 7fbe19ffb640 1 -- 172.21.15.72:0/3864036851 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3267825879 0 0) 0x7fbe08004eb0 con 0x7fbe240a9d70 2024-03-29T11:39:34.984 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.974+0000 7fbe19ffb640 1 -- 172.21.15.72:0/3864036851 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe08005300 con 0x7fbe240a9d70 2024-03-29T11:39:34.984 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.974+0000 7fbe19ffb640 1 -- 172.21.15.72:0/3864036851 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fbe14004980 con 0x7fbe240a9d70 2024-03-29T11:39:34.984 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.974+0000 7fbe19ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:34.984 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.974+0000 7fbe19ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:39:34.984 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.974+0000 7fbe19ffb640 10 monclient(hunting): dump: 2024-03-29T11:39:34.984 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:39:34.984 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:39:34.984 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:34.985 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:34.985 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:34.985 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:34.985 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:39:34.985 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:39:34.985 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:39:34.985 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:34.985 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.974+0000 7fbe19ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:34.985 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.974+0000 7fbe19ffb640 1 -- 172.21.15.72:0/3864036851 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4216215703 0 0) 0x7fbe241ccd60 con 0x7fbe2413f650 2024-03-29T11:39:34.985 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.974+0000 7fbe19ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:39:34.985 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.974+0000 7fbe19ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:34.986 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.974+0000 7fbe19ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:34.986 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.974+0000 7fbe19ffb640 10 monclient(hunting): my global_id is 4394 2024-03-29T11:39:34.986 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.974+0000 7fbe19ffb640 1 -- 172.21.15.72:0/3864036851 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe08005880 con 0x7fbe2413f650 2024-03-29T11:39:34.986 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.974+0000 7fbe19ffb640 1 -- 172.21.15.72:0/3864036851 <== mon.0 v1:172.21.15.72:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2713344521 0 0) 0x7fbe08005300 con 0x7fbe240a9d70 2024-03-29T11:39:34.986 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.974+0000 7fbe19ffb640 1 -- 172.21.15.72:0/3864036851 >> v1:172.21.15.72:6790/0 conn(0x7fbe2413f650 legacy=0x7fbe24138ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:34.986 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.974+0000 7fbe19ffb640 1 -- 172.21.15.72:0/3864036851 >> v1:172.21.15.89:6789/0 conn(0x7fbe241457f0 legacy=0x7fbe241393f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:34.986 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.974+0000 7fbe19ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:39:34.986 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.974+0000 7fbe19ffb640 1 monclient: found mon.a 2024-03-29T11:39:34.986 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.974+0000 7fbe19ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:34.986 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.974+0000 7fbe19ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:39:34.986 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.974+0000 7fbe19ffb640 1 -- 172.21.15.72:0/3864036851 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe24139b20 con 0x7fbe240a9d70 2024-03-29T11:39:34.986 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.974+0000 7fbe19ffb640 10 monclient: _finish_auth 0 2024-03-29T11:39:34.986 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.974+0000 7fbe19ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:39:34.986 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.974+0000 7fbe19ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:34.987 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.975+0000 7fbe2b889640 5 monclient: authenticate success, global_id 4392 2024-03-29T11:39:34.987 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.975+0000 7fbe2b889640 10 monclient: _renew_subs 2024-03-29T11:39:34.987 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.975+0000 7fbe2b889640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:39:34.987 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.975+0000 7fbe2b889640 1 -- 172.21.15.72:0/3864036851 --> v1:172.21.15.72:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe241cf030 con 0x7fbe240a9d70 2024-03-29T11:39:34.987 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.975+0000 7fbe2b889640 10 client.4392.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:34.987 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.975+0000 7fbe2b889640 10 monclient: _renew_subs 2024-03-29T11:39:34.987 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.975+0000 7fbe2b889640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:39:34.987 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.975+0000 7fbe2b889640 1 -- 172.21.15.72:0/3864036851 --> v1:172.21.15.72:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe240b16b0 con 0x7fbe240a9d70 2024-03-29T11:39:34.987 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.975+0000 7fbe2b889640 1 librados: init done 2024-03-29T11:39:34.987 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.975+0000 7fbe19ffb640 1 -- 172.21.15.72:0/3864036851 <== mon.0 v1:172.21.15.72:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe14001a60 con 0x7fbe240a9d70 2024-03-29T11:39:34.987 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.975+0000 7fbe19ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:34.987 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.975+0000 7fbe19ffb640 1 -- 172.21.15.72:0/3864036851 <== mon.0 v1:172.21.15.72:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fbe14004ff0 con 0x7fbe240a9d70 2024-03-29T11:39:34.987 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.975+0000 7fbe19ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:34.987 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.975+0000 7fbe19ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:34.987 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.975+0000 7fbe1affd640 4 set_mon_vals no callback set 2024-03-29T11:39:34.987 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.975+0000 7fbe19ffb640 10 monclient: dump: 2024-03-29T11:39:34.988 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:39:34.988 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:39:34.988 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:34.988 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:34.988 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:34.988 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:34.988 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:39:34.988 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:39:34.988 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:39:34.988 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:34.988 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.977+0000 7fbe2b889640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:34.988 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.977+0000 7fbe2b889640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:34.988 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.977+0000 7fbe2b889640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:39:34.988 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.977+0000 7fbe2b889640 1 -- 172.21.15.72:0/3864036851 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe04005080 con 0x7fbe240a9d70 2024-03-29T11:39:34.988 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.977+0000 7fbe19ffb640 1 -- 172.21.15.72:0/3864036851 <== mon.0 v1:172.21.15.72:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7fbe140051d0 con 0x7fbe240a9d70 2024-03-29T11:39:34.989 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.978+0000 7fbe19ffb640 1 -- 172.21.15.72:0/3864036851 <== mon.0 v1:172.21.15.72:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7fbe14084d90 con 0x7fbe240a9d70 2024-03-29T11:39:34.989 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.978+0000 7fbe19ffb640 10 client.4392.objecter ms_dispatch 0x7fbe24000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:34.989 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.978+0000 7fbe19ffb640 3 client.4392.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:34.989 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.978+0000 7fbe19ffb640 3 client.4392.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:34.989 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.978+0000 7fbe19ffb640 20 client.4392.objecter dump_active .. 0 homeless 2024-03-29T11:39:34.989 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.978+0000 7fbe28e24640 10 monclient: get_auth_request con 0x7fbe0806e020 auth_method 0 2024-03-29T11:39:34.989 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.979+0000 7fbe19ffb640 10 client.4392.objecter ms_handle_connect 0x7fbe0806e020 2024-03-29T11:39:34.990 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.987+0000 7fbe19ffb640 1 -- 172.21.15.72:0/3864036851 <== mon.0 v1:172.21.15.72:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbe14050670 con 0x7fbe240a9d70 2024-03-29T11:39:34.990 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.987+0000 7fbe19ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:34.990 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:34.987+0000 7fbe19ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:35.338 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:35.336+0000 7fbe2b889640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:35.401 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:35.336+0000 7fbe2b889640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:35.401 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:35.336+0000 7fbe2b889640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:39:35.401 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:35.336+0000 7fbe2b889640 1 -- 172.21.15.72:0/3864036851 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe04005330 con 0x7fbe240a9d70 2024-03-29T11:39:35.402 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:35.337+0000 7fbe19ffb640 1 -- 172.21.15.72:0/3864036851 <== mon.0 v1:172.21.15.72:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7fbe140525a0 con 0x7fbe240a9d70 2024-03-29T11:39:35.402 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:35.337+0000 7fbe19ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:35.402 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:35.337+0000 7fbe19ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:35.402 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:39:35.403 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:35.404 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:35.340+0000 7fbe2b889640 10 librados: watch_flush enter 2024-03-29T11:39:35.404 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:35.340+0000 7fbe2b889640 10 librados: watch_flush exit 2024-03-29T11:39:35.405 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:35.340+0000 7fbe2b889640 20 client.4392.objecter shutdown clearing up homeless session... 2024-03-29T11:39:35.405 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:35.340+0000 7fbe2b889640 10 client.4392.objecter successfully canceled tick 2024-03-29T11:39:35.405 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:35.340+0000 7fbe2b889640 1 -- 172.21.15.72:0/3864036851 >> v1:172.21.15.72:6816/1578866102 conn(0x7fbe0806e020 legacy=0x7fbe080704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:35.405 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:35.340+0000 7fbe2b889640 10 monclient: shutdown 2024-03-29T11:39:35.405 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:35.340+0000 7fbe2b889640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:35.405 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:35.340+0000 7fbe2b889640 1 -- 172.21.15.72:0/3864036851 >> v1:172.21.15.72:6789/0 conn(0x7fbe240a9d70 legacy=0x7fbe241cd390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:35.405 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:35.340+0000 7fbe2b889640 1 -- 172.21.15.72:0/3864036851 shutdown_connections 2024-03-29T11:39:35.405 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:35.340+0000 7fbe2b889640 1 -- 172.21.15.72:0/3864036851 >> 172.21.15.72:0/3864036851 conn(0x7fbe24066a00 msgr2=0x7fbe2413ac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:35.405 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:35.341+0000 7fbe2b889640 1 -- 172.21.15.72:0/3864036851 shutdown_connections 2024-03-29T11:39:35.405 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:35.341+0000 7fbe2b889640 1 -- 172.21.15.72:0/3864036851 wait complete. 2024-03-29T11:39:35.405 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:35.341+0000 7fbe2b889640 1 librados: shutdown 2024-03-29T11:39:35.406 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:42.212 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:42.347 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.346+0000 7f015173d640 10 monclient: get_monmap_and_config 2024-03-29T11:39:42.347 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.346+0000 7f015173d640 10 monclient: build_initial_monmap 2024-03-29T11:39:42.347 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.346+0000 7f015173d640 10 monclient: monmap: 2024-03-29T11:39:42.347 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:39:42.347 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:39:42.347 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:39:42.347057+0000 2024-03-29T11:39:42.348 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:39:42.347057+0000 2024-03-29T11:39:42.348 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:42.348 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:42.348 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:39:42.348 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:39:42.348 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:39:42.348 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:42.349 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.347+0000 7f015173d640 1 Processor -- start 2024-03-29T11:39:42.349 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.347+0000 7f015173d640 1 -- start start 2024-03-29T11:39:42.349 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.347+0000 7f015173d640 10 monclient: init 2024-03-29T11:39:42.349 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.347+0000 7f015173d640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:42.349 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.347+0000 7f015173d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:39:42.349 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.347+0000 7f015173d640 10 monclient(hunting): picked mon.noname-b con 0x7f014c1457f0 addr v1:172.21.15.72:6790/0 2024-03-29T11:39:42.349 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.348+0000 7f015173d640 10 monclient(hunting): picked mon.noname-c con 0x7f014c13f650 addr v1:172.21.15.89:6789/0 2024-03-29T11:39:42.349 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.348+0000 7f014affd640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f014c1457f0 0x7f014c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:58078/0 (socket says 172.21.15.72:58078) 2024-03-29T11:39:42.350 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.348+0000 7f014affd640 1 -- 172.21.15.72:0/802153955 learned_addr learned my addr 172.21.15.72:0/802153955 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:39:42.350 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.348+0000 7f015173d640 10 monclient(hunting): picked mon.noname-a con 0x7f014c0a9d70 addr v1:172.21.15.72:6789/0 2024-03-29T11:39:42.350 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.348+0000 7f015173d640 1 -- 172.21.15.72:0/802153955 --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f014c0aaef0 con 0x7f014c0a9d70 2024-03-29T11:39:42.351 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.348+0000 7f015173d640 1 -- 172.21.15.72:0/802153955 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f014c145bd0 con 0x7f014c13f650 2024-03-29T11:39:42.351 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.348+0000 7f015173d640 1 -- 172.21.15.72:0/802153955 --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f014c145eb0 con 0x7f014c1457f0 2024-03-29T11:39:42.351 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.348+0000 7f015173d640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:42.351 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.348+0000 7f015173d640 10 monclient(hunting): authenticate will time out at 1141.636805s 2024-03-29T11:39:42.351 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.349+0000 7f0149ffb640 1 -- 172.21.15.72:0/802153955 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3838943291 0 0) 0x7f014c145eb0 con 0x7f014c1457f0 2024-03-29T11:39:42.351 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.349+0000 7f0149ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:39:42.351 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.349+0000 7f0149ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:42.351 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.349+0000 7f0149ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:42.351 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.349+0000 7f0149ffb640 10 monclient(hunting): my global_id is 4397 2024-03-29T11:39:42.351 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.349+0000 7f0149ffb640 1 -- 172.21.15.72:0/802153955 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0138003520 con 0x7f014c1457f0 2024-03-29T11:39:42.351 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.349+0000 7f0149ffb640 1 -- 172.21.15.72:0/802153955 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1776183139 0 0) 0x7f0138003520 con 0x7f014c1457f0 2024-03-29T11:39:42.351 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.349+0000 7f0149ffb640 1 -- 172.21.15.72:0/802153955 >> v1:172.21.15.89:6789/0 conn(0x7f014c13f650 legacy=0x7f014c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:39:42.351 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.349+0000 7f0149ffb640 1 -- 172.21.15.72:0/802153955 >> v1:172.21.15.72:6789/0 conn(0x7f014c0a9d70 legacy=0x7f014c14d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:39:42.354 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.351+0000 7f0149ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:39:42.354 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.351+0000 7f0149ffb640 1 monclient: found mon.noname-b 2024-03-29T11:39:42.354 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.351+0000 7f0149ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:42.354 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.352+0000 7f0149ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.72:6790/0 2024-03-29T11:39:42.354 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.352+0000 7f0149ffb640 1 -- 172.21.15.72:0/802153955 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f014c146340 con 0x7f014c1457f0 2024-03-29T11:39:42.354 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.352+0000 7f0149ffb640 10 monclient: _finish_auth 0 2024-03-29T11:39:42.354 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.352+0000 7f0149ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:39:42.354 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.352+0000 7f0149ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:42.354 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.352+0000 7f0149ffb640 1 -- 172.21.15.72:0/802153955 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f013c001690 con 0x7f014c1457f0 2024-03-29T11:39:42.355 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.352+0000 7f0149ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:42.355 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.352+0000 7f0149ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:42.355 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.352+0000 7f0149ffb640 10 monclient: dump: 2024-03-29T11:39:42.355 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:39:42.355 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:39:42.355 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:42.355 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:42.355 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:42.355 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:42.355 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:39:42.355 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:39:42.355 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:39:42.355 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:42.355 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.352+0000 7f015173d640 5 monclient: authenticate success, global_id 4397 2024-03-29T11:39:42.355 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.352+0000 7f015173d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:42.356 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.353+0000 7f0149ffb640 1 -- 172.21.15.72:0/802153955 <== mon.2 v1:172.21.15.72:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f013c001860 con 0x7f014c1457f0 2024-03-29T11:39:42.356 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.353+0000 7f0149ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:42.356 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.353+0000 7f0149ffb640 1 -- 172.21.15.72:0/802153955 <== mon.2 v1:172.21.15.72:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f013c001cb0 con 0x7f014c1457f0 2024-03-29T11:39:42.356 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.353+0000 7f015173d640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:42.356 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.353+0000 7f015173d640 4 set_mon_vals no callback set 2024-03-29T11:39:42.356 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.353+0000 7f0149ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:42.356 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.353+0000 7f0149ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:42.356 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.353+0000 7f0149ffb640 10 monclient: dump: 2024-03-29T11:39:42.356 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:39:42.356 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:39:42.356 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:42.356 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:42.356 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:42.356 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:42.356 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:39:42.357 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:39:42.357 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:39:42.357 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:42.363 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.353+0000 7f015173d640 10 monclient: shutdown 2024-03-29T11:39:42.363 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.353+0000 7f015173d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:42.363 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.353+0000 7f015173d640 1 -- 172.21.15.72:0/802153955 >> v1:172.21.15.72:6790/0 conn(0x7f014c1457f0 legacy=0x7f014c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:42.363 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.353+0000 7f015173d640 1 -- 172.21.15.72:0/802153955 shutdown_connections 2024-03-29T11:39:42.363 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.353+0000 7f015173d640 1 -- 172.21.15.72:0/802153955 >> 172.21.15.72:0/802153955 conn(0x7f014c066a00 msgr2=0x7f014c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:42.363 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.354+0000 7f015173d640 1 -- 172.21.15.72:0/802153955 shutdown_connections 2024-03-29T11:39:42.363 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.354+0000 7f015173d640 1 -- 172.21.15.72:0/802153955 wait complete. 2024-03-29T11:39:42.364 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.354+0000 7f015173d640 10 monclient: build_initial_monmap 2024-03-29T11:39:42.364 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.354+0000 7f015173d640 10 monclient: monmap: 2024-03-29T11:39:42.364 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:39:42.364 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:42.364 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:39:42.364 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:39:42.364 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:42.364 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:42.364 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:39:42.364 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:39:42.364 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:39:42.364 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:42.364 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.355+0000 7f015173d640 1 librados: starting msgr at 2024-03-29T11:39:42.364 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.355+0000 7f015173d640 1 librados: starting objecter 2024-03-29T11:39:42.364 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.355+0000 7f015173d640 1 Processor -- start 2024-03-29T11:39:42.364 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.355+0000 7f015173d640 1 -- start start 2024-03-29T11:39:42.365 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.355+0000 7f015173d640 1 librados: setting wanted keys 2024-03-29T11:39:42.365 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.355+0000 7f015173d640 1 librados: calling monclient init 2024-03-29T11:39:42.365 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.355+0000 7f015173d640 10 monclient: init 2024-03-29T11:39:42.365 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.356+0000 7f015173d640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:42.365 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.356+0000 7f015173d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:39:42.365 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.356+0000 7f015173d640 10 monclient(hunting): picked mon.noname-b con 0x7f014c0a9d70 addr v1:172.21.15.89:6789/0 2024-03-29T11:39:42.365 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.356+0000 7f015173d640 10 monclient(hunting): picked mon.noname-c con 0x7f014c13f650 addr v1:172.21.15.72:6790/0 2024-03-29T11:39:42.365 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.356+0000 7f014a7fc640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f014c13f650 0x7f014c136e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:59170/0 (socket says 172.21.15.72:59170) 2024-03-29T11:39:42.365 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.356+0000 7f014a7fc640 1 -- 172.21.15.72:0/132589860 learned_addr learned my addr 172.21.15.72:0/132589860 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:39:42.365 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.356+0000 7f014affd640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f014c0a9d70 0x7f014c1da400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:35688/0 (socket says 172.21.15.72:35688) 2024-03-29T11:39:42.365 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.357+0000 7f01437fe640 10 client.?.objecter ms_handle_connect 0x7f014c13f650 2024-03-29T11:39:42.365 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.357+0000 7f01437fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:42.365 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.357+0000 7f01437fe640 10 client.?.objecter ms_handle_connect 0x7f014c0a9d70 2024-03-29T11:39:42.365 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.357+0000 7f01437fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:42.365 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.357+0000 7f015173d640 10 monclient(hunting): picked mon.noname-a con 0x7f014c1457f0 addr v1:172.21.15.72:6789/0 2024-03-29T11:39:42.365 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.357+0000 7f015173d640 1 -- 172.21.15.72:0/132589860 --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f014c0ac3f0 con 0x7f014c1457f0 2024-03-29T11:39:42.366 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.357+0000 7f015173d640 1 -- 172.21.15.72:0/132589860 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f014c1425f0 con 0x7f014c0a9d70 2024-03-29T11:39:42.366 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.357+0000 7f015173d640 1 -- 172.21.15.72:0/132589860 --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f014c1422f0 con 0x7f014c13f650 2024-03-29T11:39:42.366 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.357+0000 7f015173d640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:42.366 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.357+0000 7f015173d640 10 monclient(hunting): authenticate will time out at 1141.645626s 2024-03-29T11:39:42.366 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.357+0000 7f01437fe640 1 -- 172.21.15.72:0/132589860 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2665628372 0 0) 0x7f014c1422f0 con 0x7f014c13f650 2024-03-29T11:39:42.366 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.357+0000 7f01437fe640 10 monclient(hunting): _negotiate 2024-03-29T11:39:42.366 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.357+0000 7f01437fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:42.366 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.357+0000 7f01437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:42.366 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.357+0000 7f01437fe640 10 monclient(hunting): my global_id is 4400 2024-03-29T11:39:42.366 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.357+0000 7f01437fe640 1 -- 172.21.15.72:0/132589860 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f012c004eb0 con 0x7f014c13f650 2024-03-29T11:39:42.366 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.357+0000 7f01437fe640 1 -- 172.21.15.72:0/132589860 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4010462178 0 0) 0x7f014c1425f0 con 0x7f014c0a9d70 2024-03-29T11:39:42.366 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.357+0000 7f01437fe640 10 monclient(hunting): _negotiate 2024-03-29T11:39:42.366 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.357+0000 7f01437fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:42.366 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.357+0000 7f01437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:42.366 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.357+0000 7f01437fe640 10 monclient(hunting): my global_id is 4375 2024-03-29T11:39:42.366 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.357+0000 7f01437fe640 1 -- 172.21.15.72:0/132589860 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f014c1422f0 con 0x7f014c0a9d70 2024-03-29T11:39:42.367 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.358+0000 7f01437fe640 1 -- 172.21.15.72:0/132589860 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2146523356 0 0) 0x7f014c1422f0 con 0x7f014c0a9d70 2024-03-29T11:39:42.370 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.358+0000 7f01437fe640 1 -- 172.21.15.72:0/132589860 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f014c1425f0 con 0x7f014c0a9d70 2024-03-29T11:39:42.370 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.358+0000 7f01437fe640 1 -- 172.21.15.72:0/132589860 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f013c001b70 con 0x7f014c0a9d70 2024-03-29T11:39:42.370 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.358+0000 7f01437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:42.370 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.358+0000 7f01437fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:42.370 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.358+0000 7f01437fe640 10 monclient(hunting): dump: 2024-03-29T11:39:42.370 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:39:42.370 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:39:42.370 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:42.370 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:42.370 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:42.370 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:42.370 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:39:42.370 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:39:42.371 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:39:42.371 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:42.371 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.358+0000 7f01437fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:42.371 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.358+0000 7f01437fe640 1 -- 172.21.15.72:0/132589860 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3834360423 0 0) 0x7f012c004eb0 con 0x7f014c13f650 2024-03-29T11:39:42.371 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.358+0000 7f01437fe640 1 -- 172.21.15.72:0/132589860 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f014c1422f0 con 0x7f014c13f650 2024-03-29T11:39:42.371 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.358+0000 7f01437fe640 1 -- 172.21.15.72:0/132589860 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f0134001960 con 0x7f014c13f650 2024-03-29T11:39:42.371 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.358+0000 7f01437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:42.371 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.358+0000 7f01437fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:42.371 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.358+0000 7f01437fe640 10 monclient(hunting): dump: 2024-03-29T11:39:42.371 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:39:42.371 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:39:42.371 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:42.371 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:42.371 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:42.371 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:42.371 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:39:42.371 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:39:42.372 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:39:42.372 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:42.372 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.358+0000 7f01437fe640 1 -- 172.21.15.72:0/132589860 <== mon.1 v1:172.21.15.89:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1407251600 0 0) 0x7f014c1425f0 con 0x7f014c0a9d70 2024-03-29T11:39:42.372 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.358+0000 7f01437fe640 1 -- 172.21.15.72:0/132589860 >> v1:172.21.15.72:6790/0 conn(0x7f014c13f650 legacy=0x7f014c136e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:42.372 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.358+0000 7f01437fe640 1 -- 172.21.15.72:0/132589860 >> v1:172.21.15.72:6789/0 conn(0x7f014c1457f0 legacy=0x7f014c1e27a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:39:42.372 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.359+0000 7f01437fe640 10 monclient: _finish_hunting 0 2024-03-29T11:39:42.372 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.359+0000 7f01437fe640 1 monclient: found mon.b 2024-03-29T11:39:42.372 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.359+0000 7f01437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:42.372 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.359+0000 7f01437fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:39:42.372 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.359+0000 7f01437fe640 1 -- 172.21.15.72:0/132589860 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f014c142050 con 0x7f014c0a9d70 2024-03-29T11:39:42.372 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.359+0000 7f01437fe640 10 monclient: _finish_auth 0 2024-03-29T11:39:42.372 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.359+0000 7f01437fe640 10 monclient: _check_auth_tickets 2024-03-29T11:39:42.372 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.359+0000 7f01437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:42.372 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.359+0000 7f015173d640 5 monclient: authenticate success, global_id 4375 2024-03-29T11:39:42.372 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.359+0000 7f015173d640 10 monclient: _renew_subs 2024-03-29T11:39:42.372 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.359+0000 7f015173d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:39:42.373 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.359+0000 7f015173d640 1 -- 172.21.15.72:0/132589860 --> v1:172.21.15.89:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f014c1dab10 con 0x7f014c0a9d70 2024-03-29T11:39:42.373 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.360+0000 7f01437fe640 1 -- 172.21.15.72:0/132589860 <== mon.1 v1:172.21.15.89:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f013c004a90 con 0x7f014c0a9d70 2024-03-29T11:39:42.373 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.360+0000 7f01437fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:42.373 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.360+0000 7f01437fe640 1 -- 172.21.15.72:0/132589860 <== mon.1 v1:172.21.15.89:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f013c004da0 con 0x7f014c0a9d70 2024-03-29T11:39:42.373 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.360+0000 7f01437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:42.373 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.360+0000 7f01437fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:39:42.373 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.360+0000 7f01437fe640 10 monclient: dump: 2024-03-29T11:39:42.373 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:39:42.373 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:39:42.373 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:42.373 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:42.373 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:42.373 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:42.373 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:39:42.373 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:39:42.374 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:39:42.374 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:42.374 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.360+0000 7f015173d640 10 client.4375.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:42.374 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.360+0000 7f015173d640 10 monclient: _renew_subs 2024-03-29T11:39:42.374 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.360+0000 7f015173d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:39:42.374 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.360+0000 7f01497fa640 4 set_mon_vals no callback set 2024-03-29T11:39:42.374 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.360+0000 7f015173d640 1 -- 172.21.15.72:0/132589860 --> v1:172.21.15.89:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f014c144c70 con 0x7f014c0a9d70 2024-03-29T11:39:42.374 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.360+0000 7f015173d640 1 librados: init done 2024-03-29T11:39:42.374 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.361+0000 7f015173d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:42.374 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.361+0000 7f015173d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:42.374 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.361+0000 7f015173d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:39:42.374 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.361+0000 7f015173d640 1 -- 172.21.15.72:0/132589860 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f014c0ac250 con 0x7f014c0a9d70 2024-03-29T11:39:42.374 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.362+0000 7f01437fe640 1 -- 172.21.15.72:0/132589860 <== mon.1 v1:172.21.15.89:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f013c01ad70 con 0x7f014c0a9d70 2024-03-29T11:39:42.376 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.363+0000 7f01437fe640 1 -- 172.21.15.72:0/132589860 <== mon.1 v1:172.21.15.89:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f013c084b20 con 0x7f014c0a9d70 2024-03-29T11:39:42.376 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.363+0000 7f01437fe640 10 client.4375.objecter ms_dispatch 0x7f014c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:42.376 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.363+0000 7f01437fe640 3 client.4375.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:42.376 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.363+0000 7f01437fe640 3 client.4375.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:42.376 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.363+0000 7f01437fe640 20 client.4375.objecter dump_active .. 0 homeless 2024-03-29T11:39:42.376 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.363+0000 7f014a7fc640 10 monclient: get_auth_request con 0x7f012c06dde0 auth_method 0 2024-03-29T11:39:42.376 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.363+0000 7f01437fe640 10 client.4375.objecter ms_handle_connect 0x7f012c06dde0 2024-03-29T11:39:42.376 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.370+0000 7f01437fe640 1 -- 172.21.15.72:0/132589860 <== mon.1 v1:172.21.15.89:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f014c0ac250 con 0x7f014c0a9d70 2024-03-29T11:39:42.376 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.370+0000 7f01437fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:42.376 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.370+0000 7f01437fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:42.676 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.674+0000 7f015173d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:42.676 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.674+0000 7f015173d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:42.676 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.674+0000 7f015173d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:39:42.676 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.674+0000 7f015173d640 1 -- 172.21.15.72:0/132589860 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f014c0ac250 con 0x7f014c0a9d70 2024-03-29T11:39:42.677 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.676+0000 7f01437fe640 1 -- 172.21.15.72:0/132589860 <== mon.1 v1:172.21.15.89:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f014c0ac250 con 0x7f014c0a9d70 2024-03-29T11:39:42.677 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.676+0000 7f01437fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:42.677 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.676+0000 7f01437fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:42.678 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:39:42.678 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:42.681 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.679+0000 7f015173d640 10 librados: watch_flush enter 2024-03-29T11:39:42.681 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.679+0000 7f015173d640 10 librados: watch_flush exit 2024-03-29T11:39:42.681 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.679+0000 7f015173d640 20 client.4375.objecter shutdown clearing up homeless session... 2024-03-29T11:39:42.681 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.679+0000 7f015173d640 10 client.4375.objecter successfully canceled tick 2024-03-29T11:39:42.681 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.679+0000 7f015173d640 1 -- 172.21.15.72:0/132589860 >> v1:172.21.15.72:6816/1578866102 conn(0x7f012c06dde0 legacy=0x7f012c0702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:42.681 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.679+0000 7f015173d640 10 monclient: shutdown 2024-03-29T11:39:42.681 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.679+0000 7f015173d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:42.681 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.679+0000 7f015173d640 1 -- 172.21.15.72:0/132589860 >> v1:172.21.15.89:6789/0 conn(0x7f014c0a9d70 legacy=0x7f014c1da400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:42.682 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.679+0000 7f015173d640 1 -- 172.21.15.72:0/132589860 shutdown_connections 2024-03-29T11:39:42.682 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.679+0000 7f015173d640 1 -- 172.21.15.72:0/132589860 >> 172.21.15.72:0/132589860 conn(0x7f014c066a00 msgr2=0x7f014c0b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:42.682 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.679+0000 7f015173d640 1 -- 172.21.15.72:0/132589860 shutdown_connections 2024-03-29T11:39:42.682 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.679+0000 7f015173d640 1 -- 172.21.15.72:0/132589860 wait complete. 2024-03-29T11:39:42.682 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:42.680+0000 7f015173d640 1 librados: shutdown 2024-03-29T11:39:42.696 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:49.503 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:49.653 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.651+0000 7f4e74425640 10 monclient: get_monmap_and_config 2024-03-29T11:39:49.653 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.651+0000 7f4e74425640 10 monclient: build_initial_monmap 2024-03-29T11:39:49.654 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.651+0000 7f4e74425640 10 monclient: monmap: 2024-03-29T11:39:49.654 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:39:49.654 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:39:49.654 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:39:49.652667+0000 2024-03-29T11:39:49.654 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:39:49.652667+0000 2024-03-29T11:39:49.654 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:49.654 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:49.654 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:39:49.654 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:39:49.654 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:39:49.654 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:49.655 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.653+0000 7f4e74425640 1 Processor -- start 2024-03-29T11:39:49.655 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.653+0000 7f4e74425640 1 -- start start 2024-03-29T11:39:49.655 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.653+0000 7f4e74425640 10 monclient: init 2024-03-29T11:39:49.656 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.654+0000 7f4e74425640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:49.656 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.654+0000 7f4e74425640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:39:49.656 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.654+0000 7f4e74425640 10 monclient(hunting): picked mon.noname-b con 0x7f4e64006410 addr v1:172.21.15.72:6790/0 2024-03-29T11:39:49.656 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.654+0000 7f4e74425640 10 monclient(hunting): picked mon.noname-c con 0x7f4e6402a640 addr v1:172.21.15.89:6789/0 2024-03-29T11:39:49.656 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.654+0000 7f4e74425640 10 monclient(hunting): picked mon.noname-a con 0x7f4e640327d0 addr v1:172.21.15.72:6789/0 2024-03-29T11:39:49.656 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.654+0000 7f4e74425640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e6402c820 con 0x7f4e640327d0 2024-03-29T11:39:49.656 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.654+0000 7f4e74425640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e6402a120 con 0x7f4e6402a640 2024-03-29T11:39:49.656 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.654+0000 7f4e74425640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e640c75b0 con 0x7f4e64006410 2024-03-29T11:39:49.656 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.654+0000 7f4e74425640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:49.656 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.654+0000 7f4e74425640 10 monclient(hunting): authenticate will time out at 1148.942268s 2024-03-29T11:39:49.656 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.654+0000 7f4e721c1640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f4e64006410 0x7f4e640c6ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:40968/0 (socket says 172.21.15.72:40968) 2024-03-29T11:39:49.656 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.654+0000 7f4e721c1640 1 -- 172.21.15.72:0/4179025192 learned_addr learned my addr 172.21.15.72:0/4179025192 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:39:49.657 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.655+0000 7f4e711bf640 1 -- 172.21.15.72:0/4179025192 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 249601241 0 0) 0x7f4e640c75b0 con 0x7f4e64006410 2024-03-29T11:39:49.657 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.655+0000 7f4e711bf640 10 monclient(hunting): _negotiate 2024-03-29T11:39:49.657 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.655+0000 7f4e711bf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:49.657 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.655+0000 7f4e711bf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:49.657 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.655+0000 7f4e711bf640 10 monclient(hunting): my global_id is 4403 2024-03-29T11:39:49.658 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.655+0000 7f4e711bf640 1 -- 172.21.15.72:0/4179025192 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e58003520 con 0x7f4e64006410 2024-03-29T11:39:49.658 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.657+0000 7f4e711bf640 1 -- 172.21.15.72:0/4179025192 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1530276433 0 0) 0x7f4e58003520 con 0x7f4e64006410 2024-03-29T11:39:49.659 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.657+0000 7f4e711bf640 1 -- 172.21.15.72:0/4179025192 >> v1:172.21.15.89:6789/0 conn(0x7f4e6402a640 legacy=0x7f4e640320c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:39:49.659 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.657+0000 7f4e711bf640 1 -- 172.21.15.72:0/4179025192 >> v1:172.21.15.72:6789/0 conn(0x7f4e640327d0 legacy=0x7f4e640c8200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:39:49.662 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.660+0000 7f4e711bf640 10 monclient: _finish_hunting 0 2024-03-29T11:39:49.663 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.660+0000 7f4e711bf640 1 monclient: found mon.noname-b 2024-03-29T11:39:49.663 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.660+0000 7f4e711bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:49.663 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.660+0000 7f4e711bf640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.72:6790/0 2024-03-29T11:39:49.663 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.660+0000 7f4e711bf640 1 -- 172.21.15.72:0/4179025192 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e64033110 con 0x7f4e64006410 2024-03-29T11:39:49.663 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.660+0000 7f4e711bf640 10 monclient: _finish_auth 0 2024-03-29T11:39:49.663 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.660+0000 7f4e711bf640 10 monclient: _check_auth_tickets 2024-03-29T11:39:49.663 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.660+0000 7f4e711bf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:49.663 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.660+0000 7f4e711bf640 1 -- 172.21.15.72:0/4179025192 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f4e5c001690 con 0x7f4e64006410 2024-03-29T11:39:49.663 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.660+0000 7f4e711bf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:49.663 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.660+0000 7f4e711bf640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:49.663 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.660+0000 7f4e711bf640 10 monclient: dump: 2024-03-29T11:39:49.663 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:39:49.663 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:39:49.663 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:49.663 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:49.664 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:49.664 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:49.664 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:39:49.664 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:39:49.664 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:39:49.664 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:49.664 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.661+0000 7f4e74425640 5 monclient: authenticate success, global_id 4403 2024-03-29T11:39:49.664 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.661+0000 7f4e74425640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:49.664 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.661+0000 7f4e711bf640 1 -- 172.21.15.72:0/4179025192 <== mon.2 v1:172.21.15.72:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e5c001860 con 0x7f4e64006410 2024-03-29T11:39:49.664 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.661+0000 7f4e711bf640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:49.665 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.661+0000 7f4e74425640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:49.665 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.661+0000 7f4e74425640 4 set_mon_vals no callback set 2024-03-29T11:39:49.665 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.662+0000 7f4e74425640 10 monclient: shutdown 2024-03-29T11:39:49.665 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.662+0000 7f4e74425640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:49.665 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.662+0000 7f4e74425640 1 -- 172.21.15.72:0/4179025192 >> v1:172.21.15.72:6790/0 conn(0x7f4e64006410 legacy=0x7f4e640c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:49.666 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.665+0000 7f4e74425640 1 -- 172.21.15.72:0/4179025192 shutdown_connections 2024-03-29T11:39:49.666 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.665+0000 7f4e74425640 1 -- 172.21.15.72:0/4179025192 >> 172.21.15.72:0/4179025192 conn(0x7f4e640058f0 msgr2=0x7f4e64005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:49.666 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.665+0000 7f4e74425640 1 -- 172.21.15.72:0/4179025192 shutdown_connections 2024-03-29T11:39:49.666 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.665+0000 7f4e74425640 1 -- 172.21.15.72:0/4179025192 wait complete. 2024-03-29T11:39:49.668 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.666+0000 7f4e74425640 10 monclient: build_initial_monmap 2024-03-29T11:39:49.668 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.666+0000 7f4e74425640 10 monclient: monmap: 2024-03-29T11:39:49.668 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:39:49.668 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:49.668 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:39:49.668 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:39:49.668 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:49.668 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:49.668 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:39:49.668 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:39:49.668 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:39:49.668 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:49.668 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.666+0000 7f4e74425640 1 librados: starting msgr at 2024-03-29T11:39:49.669 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.666+0000 7f4e74425640 1 librados: starting objecter 2024-03-29T11:39:49.669 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.666+0000 7f4e74425640 1 Processor -- start 2024-03-29T11:39:49.669 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.667+0000 7f4e74425640 1 -- start start 2024-03-29T11:39:49.669 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.667+0000 7f4e74425640 1 librados: setting wanted keys 2024-03-29T11:39:49.669 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.667+0000 7f4e74425640 1 librados: calling monclient init 2024-03-29T11:39:49.669 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.667+0000 7f4e74425640 10 monclient: init 2024-03-29T11:39:49.669 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.668+0000 7f4e74425640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:49.669 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.668+0000 7f4e74425640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:39:49.670 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.668+0000 7f4e74425640 10 monclient(hunting): picked mon.noname-a con 0x7f4e64006410 addr v1:172.21.15.72:6789/0 2024-03-29T11:39:49.670 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.668+0000 7f4e74425640 10 monclient(hunting): picked mon.noname-b con 0x7f4e6402a640 addr v1:172.21.15.89:6789/0 2024-03-29T11:39:49.670 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.668+0000 7f4e74425640 10 monclient(hunting): picked mon.noname-c con 0x7f4e640327d0 addr v1:172.21.15.72:6790/0 2024-03-29T11:39:49.670 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.668+0000 7f4e74425640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e640d8980 con 0x7f4e64006410 2024-03-29T11:39:49.670 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.668+0000 7f4e74425640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e640da4a0 con 0x7f4e6402a640 2024-03-29T11:39:49.670 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.669+0000 7f4e74425640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e640d9590 con 0x7f4e640327d0 2024-03-29T11:39:49.670 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.669+0000 7f4e74425640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:49.670 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.669+0000 7f4e74425640 10 monclient(hunting): authenticate will time out at 1148.957049s 2024-03-29T11:39:49.672 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.670+0000 7f4e729c2640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f4e640327d0 0x7f4e6416be60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:33696/0 (socket says 172.21.15.72:33696) 2024-03-29T11:39:49.672 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.670+0000 7f4e729c2640 1 -- 172.21.15.72:0/849339720 learned_addr learned my addr 172.21.15.72:0/849339720 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:39:49.672 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.670+0000 7f4e62ffd640 10 client.?.objecter ms_handle_connect 0x7f4e640327d0 2024-03-29T11:39:49.672 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.670+0000 7f4e62ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:49.672 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.670+0000 7f4e62ffd640 1 -- 172.21.15.72:0/849339720 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 790018528 0 0) 0x7f4e640d9590 con 0x7f4e640327d0 2024-03-29T11:39:49.673 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.670+0000 7f4e62ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:49.673 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.670+0000 7f4e62ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:49.673 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.670+0000 7f4e62ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:49.673 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.670+0000 7f4e62ffd640 10 monclient(hunting): my global_id is 4406 2024-03-29T11:39:49.673 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.670+0000 7f4e62ffd640 1 -- 172.21.15.72:0/849339720 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e50004eb0 con 0x7f4e640327d0 2024-03-29T11:39:49.673 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.670+0000 7f4e62ffd640 1 -- 172.21.15.72:0/849339720 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2374969388 0 0) 0x7f4e50004eb0 con 0x7f4e640327d0 2024-03-29T11:39:49.673 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.670+0000 7f4e62ffd640 1 -- 172.21.15.72:0/849339720 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e640d9590 con 0x7f4e640327d0 2024-03-29T11:39:49.673 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.670+0000 7f4e62ffd640 1 -- 172.21.15.72:0/849339720 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f4e6c04fdd0 con 0x7f4e640327d0 2024-03-29T11:39:49.673 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.670+0000 7f4e62ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:49.673 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.670+0000 7f4e62ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:49.673 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.670+0000 7f4e62ffd640 10 monclient(hunting): dump: 2024-03-29T11:39:49.673 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:39:49.673 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:39:49.673 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:49.673 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:49.673 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:49.673 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:49.674 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:39:49.674 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:39:49.674 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:39:49.674 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:49.674 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.670+0000 7f4e62ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:49.674 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.672+0000 7f4e62ffd640 1 -- 172.21.15.72:0/849339720 <== mon.2 v1:172.21.15.72:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3353557334 0 0) 0x7f4e640d9590 con 0x7f4e640327d0 2024-03-29T11:39:49.674 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.672+0000 7f4e62ffd640 1 -- 172.21.15.72:0/849339720 >> v1:172.21.15.89:6789/0 conn(0x7f4e6402a640 legacy=0x7f4e64168730 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:39:49.677 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.675+0000 7f4e62ffd640 1 -- 172.21.15.72:0/849339720 >> v1:172.21.15.72:6789/0 conn(0x7f4e64006410 legacy=0x7f4e640d8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:49.677 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.675+0000 7f4e62ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:39:49.677 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.675+0000 7f4e62ffd640 1 monclient: found mon.c 2024-03-29T11:39:49.677 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.675+0000 7f4e62ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:49.677 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.675+0000 7f4e62ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:39:49.677 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.675+0000 7f4e62ffd640 1 -- 172.21.15.72:0/849339720 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e640d7640 con 0x7f4e640327d0 2024-03-29T11:39:49.677 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.675+0000 7f4e62ffd640 10 monclient: _finish_auth 0 2024-03-29T11:39:49.677 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.675+0000 7f4e62ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:39:49.677 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.675+0000 7f4e62ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:49.677 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.675+0000 7f4e62ffd640 10 client.?.objecter ms_handle_connect 0x7f4e64006410 2024-03-29T11:39:49.677 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.675+0000 7f4e62ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:49.677 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.675+0000 7f4e74425640 5 monclient: authenticate success, global_id 4406 2024-03-29T11:39:49.678 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.676+0000 7f4e74425640 10 monclient: _renew_subs 2024-03-29T11:39:49.678 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.676+0000 7f4e74425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:39:49.678 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.676+0000 7f4e74425640 1 -- 172.21.15.72:0/849339720 --> v1:172.21.15.72:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e640d5ec0 con 0x7f4e640327d0 2024-03-29T11:39:49.678 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.676+0000 7f4e74425640 10 client.4406.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:49.678 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.676+0000 7f4e74425640 10 monclient: _renew_subs 2024-03-29T11:39:49.678 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.676+0000 7f4e74425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:39:49.678 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.676+0000 7f4e74425640 1 -- 172.21.15.72:0/849339720 --> v1:172.21.15.72:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e640c4310 con 0x7f4e640327d0 2024-03-29T11:39:49.678 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.676+0000 7f4e74425640 1 librados: init done 2024-03-29T11:39:49.678 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.676+0000 7f4e62ffd640 1 -- 172.21.15.72:0/849339720 <== mon.2 v1:172.21.15.72:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e6c04f400 con 0x7f4e640327d0 2024-03-29T11:39:49.678 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.676+0000 7f4e62ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:49.678 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.676+0000 7f4e62ffd640 1 -- 172.21.15.72:0/849339720 <== mon.2 v1:172.21.15.72:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f4e6c075f10 con 0x7f4e640327d0 2024-03-29T11:39:49.678 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.676+0000 7f4e62ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:49.678 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.676+0000 7f4e62ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:49.678 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.676+0000 7f4e62ffd640 10 monclient: dump: 2024-03-29T11:39:49.678 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:39:49.678 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:39:49.678 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:49.679 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:49.679 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:49.679 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:49.679 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:39:49.679 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:39:49.679 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:39:49.679 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:49.679 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.676+0000 7f4e709be640 4 set_mon_vals no callback set 2024-03-29T11:39:49.680 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.677+0000 7f4e74425640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:49.681 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.678+0000 7f4e62ffd640 1 -- 172.21.15.72:0/849339720 <== mon.2 v1:172.21.15.72:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f4e6c067bf0 con 0x7f4e640327d0 2024-03-29T11:39:49.681 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.678+0000 7f4e62ffd640 1 -- 172.21.15.72:0/849339720 <== mon.2 v1:172.21.15.72:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f4e6c0ff470 con 0x7f4e640327d0 2024-03-29T11:39:49.681 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.678+0000 7f4e62ffd640 10 client.4406.objecter ms_dispatch 0x7f4e6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:49.681 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.678+0000 7f4e62ffd640 3 client.4406.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:49.681 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.678+0000 7f4e62ffd640 3 client.4406.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:49.681 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.678+0000 7f4e62ffd640 20 client.4406.objecter dump_active .. 0 homeless 2024-03-29T11:39:49.681 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.679+0000 7f4e74425640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:49.681 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.679+0000 7f4e74425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:39:49.681 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.679+0000 7f4e74425640 1 -- 172.21.15.72:0/849339720 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e54005080 con 0x7f4e640327d0 2024-03-29T11:39:49.681 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.680+0000 7f4e721c1640 10 monclient: get_auth_request con 0x7f4e5006dac0 auth_method 0 2024-03-29T11:39:49.682 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.680+0000 7f4e62ffd640 10 client.4406.objecter ms_handle_connect 0x7f4e5006dac0 2024-03-29T11:39:49.690 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.689+0000 7f4e62ffd640 1 -- 172.21.15.72:0/849339720 <== mon.2 v1:172.21.15.72:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4e6c0cad50 con 0x7f4e640327d0 2024-03-29T11:39:49.690 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.689+0000 7f4e62ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:49.690 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:49.689+0000 7f4e62ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:50.019 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:50.017+0000 7f4e74425640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:50.019 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:50.017+0000 7f4e74425640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:50.019 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:50.017+0000 7f4e74425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:39:50.019 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:50.017+0000 7f4e74425640 1 -- 172.21.15.72:0/849339720 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e54005330 con 0x7f4e640327d0 2024-03-29T11:39:50.019 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:50.018+0000 7f4e62ffd640 1 -- 172.21.15.72:0/849339720 <== mon.2 v1:172.21.15.72:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f4e6c0ccc80 con 0x7f4e640327d0 2024-03-29T11:39:50.019 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:50.018+0000 7f4e62ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:50.019 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:50.018+0000 7f4e62ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:50.020 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:39:50.020 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:50.023 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:50.022+0000 7f4e74425640 10 librados: watch_flush enter 2024-03-29T11:39:50.023 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:50.022+0000 7f4e74425640 10 librados: watch_flush exit 2024-03-29T11:39:50.023 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:50.022+0000 7f4e74425640 20 client.4406.objecter shutdown clearing up homeless session... 2024-03-29T11:39:50.023 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:50.022+0000 7f4e74425640 10 client.4406.objecter successfully canceled tick 2024-03-29T11:39:50.024 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:50.022+0000 7f4e74425640 1 -- 172.21.15.72:0/849339720 >> v1:172.21.15.72:6816/1578866102 conn(0x7f4e5006dac0 legacy=0x7f4e5006ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:50.024 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:50.022+0000 7f4e74425640 10 monclient: shutdown 2024-03-29T11:39:50.024 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:50.022+0000 7f4e74425640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:50.024 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:50.022+0000 7f4e74425640 1 -- 172.21.15.72:0/849339720 >> v1:172.21.15.72:6790/0 conn(0x7f4e640327d0 legacy=0x7f4e6416be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:50.024 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:50.022+0000 7f4e74425640 1 -- 172.21.15.72:0/849339720 shutdown_connections 2024-03-29T11:39:50.024 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:50.022+0000 7f4e74425640 1 -- 172.21.15.72:0/849339720 >> 172.21.15.72:0/849339720 conn(0x7f4e640058f0 msgr2=0x7f4e640043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:50.026 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:50.025+0000 7f4e74425640 1 -- 172.21.15.72:0/849339720 shutdown_connections 2024-03-29T11:39:50.026 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:50.025+0000 7f4e74425640 1 -- 172.21.15.72:0/849339720 wait complete. 2024-03-29T11:39:50.027 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:50.026+0000 7f4e74425640 1 librados: shutdown 2024-03-29T11:39:50.044 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:56.851 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:57.004 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.002+0000 7fe8bcdf6640 10 monclient: get_monmap_and_config 2024-03-29T11:39:57.005 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.002+0000 7fe8bcdf6640 10 monclient: build_initial_monmap 2024-03-29T11:39:57.005 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.002+0000 7fe8bcdf6640 10 monclient: monmap: 2024-03-29T11:39:57.005 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:39:57.006 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:39:57.006 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:39:57.003906+0000 2024-03-29T11:39:57.006 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:39:57.003906+0000 2024-03-29T11:39:57.006 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:57.006 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:57.006 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:39:57.006 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:39:57.006 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:39:57.006 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:57.007 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.005+0000 7fe8bcdf6640 1 Processor -- start 2024-03-29T11:39:57.007 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.005+0000 7fe8bcdf6640 1 -- start start 2024-03-29T11:39:57.007 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.005+0000 7fe8bcdf6640 10 monclient: init 2024-03-29T11:39:57.011 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.007+0000 7fe8bcdf6640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:57.011 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.007+0000 7fe8bcdf6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:39:57.011 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.007+0000 7fe8bcdf6640 10 monclient(hunting): picked mon.noname-b con 0x7fe8b81422b0 addr v1:172.21.15.72:6790/0 2024-03-29T11:39:57.011 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.007+0000 7fe8bcdf6640 10 monclient(hunting): picked mon.noname-c con 0x7fe8b8142690 addr v1:172.21.15.89:6789/0 2024-03-29T11:39:57.012 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.007+0000 7fe8bcdf6640 10 monclient(hunting): picked mon.noname-a con 0x7fe8b80b17d0 addr v1:172.21.15.72:6789/0 2024-03-29T11:39:57.012 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.007+0000 7fe8bcdf6640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8b8142c80 con 0x7fe8b80b17d0 2024-03-29T11:39:57.012 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.007+0000 7fe8bcdf6640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8b80aae40 con 0x7fe8b8142690 2024-03-29T11:39:57.012 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.007+0000 7fe8bcdf6640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8b813fbd0 con 0x7fe8b81422b0 2024-03-29T11:39:57.012 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.007+0000 7fe8bcdf6640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:57.012 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.007+0000 7fe8bcdf6640 10 monclient(hunting): authenticate will time out at 1156.295125s 2024-03-29T11:39:57.012 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.010+0000 7fe8b6d9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fe8b80b17d0 0x7fe8b80b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:39395/0 (socket says 172.21.15.72:39395) 2024-03-29T11:39:57.012 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.010+0000 7fe8b6d9d640 1 -- 172.21.15.72:0/2008694903 learned_addr learned my addr 172.21.15.72:0/2008694903 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:39:57.019 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.011+0000 7fe8b559a640 1 -- 172.21.15.72:0/2008694903 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1369452699 0 0) 0x7fe8b8142c80 con 0x7fe8b80b17d0 2024-03-29T11:39:57.019 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.011+0000 7fe8b559a640 10 monclient(hunting): _negotiate 2024-03-29T11:39:57.019 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.011+0000 7fe8b559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:57.019 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.011+0000 7fe8b559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:57.019 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.011+0000 7fe8b559a640 10 monclient(hunting): my global_id is 4401 2024-03-29T11:39:57.020 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.011+0000 7fe8b559a640 1 -- 172.21.15.72:0/2008694903 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8a0003520 con 0x7fe8b80b17d0 2024-03-29T11:39:57.020 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.011+0000 7fe8b559a640 1 -- 172.21.15.72:0/2008694903 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3338609896 0 0) 0x7fe8b813fbd0 con 0x7fe8b81422b0 2024-03-29T11:39:57.020 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.011+0000 7fe8b559a640 10 monclient(hunting): _negotiate 2024-03-29T11:39:57.020 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.011+0000 7fe8b559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:57.020 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.011+0000 7fe8b559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:57.020 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.011+0000 7fe8b559a640 10 monclient(hunting): my global_id is 4409 2024-03-29T11:39:57.020 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.011+0000 7fe8b559a640 1 -- 172.21.15.72:0/2008694903 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8b8142c80 con 0x7fe8b81422b0 2024-03-29T11:39:57.020 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.011+0000 7fe8b559a640 1 -- 172.21.15.72:0/2008694903 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 401333728 0 0) 0x7fe8a0003520 con 0x7fe8b80b17d0 2024-03-29T11:39:57.020 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.012+0000 7fe8b559a640 1 -- 172.21.15.72:0/2008694903 >> v1:172.21.15.72:6790/0 conn(0x7fe8b81422b0 legacy=0x7fe8b813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:57.020 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.012+0000 7fe8b559a640 1 -- 172.21.15.72:0/2008694903 >> v1:172.21.15.89:6789/0 conn(0x7fe8b8142690 legacy=0x7fe8b80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:39:57.020 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.012+0000 7fe8b559a640 10 monclient: _finish_hunting 0 2024-03-29T11:39:57.020 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.012+0000 7fe8b559a640 1 monclient: found mon.noname-a 2024-03-29T11:39:57.020 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.012+0000 7fe8b559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:57.020 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.012+0000 7fe8b559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.72:6789/0 2024-03-29T11:39:57.020 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.012+0000 7fe8b559a640 1 -- 172.21.15.72:0/2008694903 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b8151940 con 0x7fe8b80b17d0 2024-03-29T11:39:57.020 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.012+0000 7fe8b559a640 10 monclient: _finish_auth 0 2024-03-29T11:39:57.020 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.012+0000 7fe8b559a640 10 monclient: _check_auth_tickets 2024-03-29T11:39:57.021 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.012+0000 7fe8b559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:57.021 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.012+0000 7fe8b559a640 1 -- 172.21.15.72:0/2008694903 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fe8ac001690 con 0x7fe8b80b17d0 2024-03-29T11:39:57.021 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.012+0000 7fe8b559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:57.021 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.012+0000 7fe8b559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:39:57.021 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.012+0000 7fe8b559a640 10 monclient: dump: 2024-03-29T11:39:57.021 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:39:57.021 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:39:57.021 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:57.021 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:57.021 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:57.021 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:57.021 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:39:57.021 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:39:57.021 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:39:57.021 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:57.021 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.012+0000 7fe8b559a640 1 -- 172.21.15.72:0/2008694903 <== mon.0 v1:172.21.15.72:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8ac001870 con 0x7fe8b80b17d0 2024-03-29T11:39:57.021 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.012+0000 7fe8b559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:57.022 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.012+0000 7fe8b559a640 1 -- 172.21.15.72:0/2008694903 <== mon.0 v1:172.21.15.72:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fe8ac001690 con 0x7fe8b80b17d0 2024-03-29T11:39:57.022 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.012+0000 7fe8b559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:57.022 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.012+0000 7fe8b559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:57.022 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.012+0000 7fe8b559a640 10 monclient: dump: 2024-03-29T11:39:57.022 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:39:57.022 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:39:57.022 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:57.022 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:57.022 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:57.022 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:57.022 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:39:57.022 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:39:57.022 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:39:57.022 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:57.022 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.013+0000 7fe8bcdf6640 5 monclient: authenticate success, global_id 4401 2024-03-29T11:39:57.023 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.013+0000 7fe8bcdf6640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:57.023 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.013+0000 7fe8bcdf6640 4 set_mon_vals no callback set 2024-03-29T11:39:57.023 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.014+0000 7fe8bcdf6640 10 monclient: shutdown 2024-03-29T11:39:57.023 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.014+0000 7fe8bcdf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:57.023 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.014+0000 7fe8bcdf6640 1 -- 172.21.15.72:0/2008694903 >> v1:172.21.15.72:6789/0 conn(0x7fe8b80b17d0 legacy=0x7fe8b80b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:57.023 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.016+0000 7fe8bcdf6640 1 -- 172.21.15.72:0/2008694903 shutdown_connections 2024-03-29T11:39:57.023 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.016+0000 7fe8bcdf6640 1 -- 172.21.15.72:0/2008694903 >> 172.21.15.72:0/2008694903 conn(0x7fe8b8066a00 msgr2=0x7fe8b8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:57.023 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.016+0000 7fe8bcdf6640 1 -- 172.21.15.72:0/2008694903 shutdown_connections 2024-03-29T11:39:57.023 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.016+0000 7fe8bcdf6640 1 -- 172.21.15.72:0/2008694903 wait complete. 2024-03-29T11:39:57.023 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.018+0000 7fe8bcdf6640 10 monclient: build_initial_monmap 2024-03-29T11:39:57.023 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.018+0000 7fe8bcdf6640 10 monclient: monmap: 2024-03-29T11:39:57.023 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:39:57.023 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:57.023 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:39:57.023 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:39:57.023 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:57.023 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:57.024 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:39:57.024 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:39:57.024 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:39:57.024 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:57.024 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.018+0000 7fe8bcdf6640 1 librados: starting msgr at 2024-03-29T11:39:57.024 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.018+0000 7fe8bcdf6640 1 librados: starting objecter 2024-03-29T11:39:57.024 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.018+0000 7fe8bcdf6640 1 Processor -- start 2024-03-29T11:39:57.025 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.020+0000 7fe8bcdf6640 1 -- start start 2024-03-29T11:39:57.025 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.020+0000 7fe8bcdf6640 1 librados: setting wanted keys 2024-03-29T11:39:57.025 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.020+0000 7fe8bcdf6640 1 librados: calling monclient init 2024-03-29T11:39:57.032 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.020+0000 7fe8bcdf6640 10 monclient: init 2024-03-29T11:39:57.032 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.021+0000 7fe8bcdf6640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:57.032 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.021+0000 7fe8bcdf6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:39:57.032 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.021+0000 7fe8bcdf6640 10 monclient(hunting): picked mon.noname-b con 0x7fe8b80b17d0 addr v1:172.21.15.89:6789/0 2024-03-29T11:39:57.032 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.021+0000 7fe8bcdf6640 10 monclient(hunting): picked mon.noname-c con 0x7fe8b81422b0 addr v1:172.21.15.72:6790/0 2024-03-29T11:39:57.032 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.021+0000 7fe8bcdf6640 10 monclient(hunting): picked mon.noname-a con 0x7fe8b8142690 addr v1:172.21.15.72:6789/0 2024-03-29T11:39:57.032 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.021+0000 7fe8bcdf6640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8b806f1c0 con 0x7fe8b8142690 2024-03-29T11:39:57.032 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.021+0000 7fe8bcdf6640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8b812dc00 con 0x7fe8b80b17d0 2024-03-29T11:39:57.032 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.021+0000 7fe8bcdf6640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8b806f3f0 con 0x7fe8b81422b0 2024-03-29T11:39:57.033 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.021+0000 7fe8bcdf6640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:57.033 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.021+0000 7fe8bcdf6640 10 monclient(hunting): authenticate will time out at 1156.309408s 2024-03-29T11:39:57.033 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.028+0000 7fe8b659c640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fe8b80b17d0 0x7fe8b8139400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:50837/0 (socket says 172.21.15.72:50837) 2024-03-29T11:39:57.033 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.028+0000 7fe8b659c640 1 -- 172.21.15.72:0/450285980 learned_addr learned my addr 172.21.15.72:0/450285980 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:39:57.033 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.028+0000 7fe8b5d9b640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7fe8b81422b0 0x7fe8b814d950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:57925/0 (socket says 172.21.15.72:57925) 2024-03-29T11:39:57.033 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.029+0000 7fe8a6ffd640 10 client.?.objecter ms_handle_connect 0x7fe8b81422b0 2024-03-29T11:39:57.033 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.029+0000 7fe8a6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:57.033 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.029+0000 7fe8a6ffd640 10 client.?.objecter ms_handle_connect 0x7fe8b80b17d0 2024-03-29T11:39:57.033 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.029+0000 7fe8a6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:57.033 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.029+0000 7fe8a6ffd640 1 -- 172.21.15.72:0/450285980 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2919736793 0 0) 0x7fe8b806f3f0 con 0x7fe8b81422b0 2024-03-29T11:39:57.033 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.029+0000 7fe8a6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:57.033 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.029+0000 7fe8a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:57.033 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.029+0000 7fe8a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:57.033 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.029+0000 7fe8a6ffd640 10 monclient(hunting): my global_id is 4412 2024-03-29T11:39:57.033 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.029+0000 7fe8a6ffd640 1 -- 172.21.15.72:0/450285980 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe898004eb0 con 0x7fe8b81422b0 2024-03-29T11:39:57.033 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.029+0000 7fe8a6ffd640 1 -- 172.21.15.72:0/450285980 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2255100619 0 0) 0x7fe898004eb0 con 0x7fe8b81422b0 2024-03-29T11:39:57.034 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.029+0000 7fe8a6ffd640 1 -- 172.21.15.72:0/450285980 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8b806f3f0 con 0x7fe8b81422b0 2024-03-29T11:39:57.034 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.029+0000 7fe8a6ffd640 1 -- 172.21.15.72:0/450285980 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fe8a80017d0 con 0x7fe8b81422b0 2024-03-29T11:39:57.034 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.029+0000 7fe8a6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:57.034 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.029+0000 7fe8a6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:57.034 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.029+0000 7fe8a6ffd640 10 monclient(hunting): dump: 2024-03-29T11:39:57.034 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:39:57.034 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:39:57.034 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:57.034 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:57.034 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:57.034 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:57.034 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:39:57.034 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:39:57.034 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:39:57.034 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:57.034 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.029+0000 7fe8a6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:57.035 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.030+0000 7fe8a6ffd640 1 -- 172.21.15.72:0/450285980 <== mon.2 v1:172.21.15.72:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3562565725 0 0) 0x7fe8b806f3f0 con 0x7fe8b81422b0 2024-03-29T11:39:57.035 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.030+0000 7fe8a6ffd640 1 -- 172.21.15.72:0/450285980 >> v1:172.21.15.89:6789/0 conn(0x7fe8b80b17d0 legacy=0x7fe8b8139400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:57.035 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.030+0000 7fe8a6ffd640 1 -- 172.21.15.72:0/450285980 >> v1:172.21.15.72:6789/0 conn(0x7fe8b8142690 legacy=0x7fe8b81dea40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:39:57.035 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.033+0000 7fe8a6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:39:57.035 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.033+0000 7fe8a6ffd640 1 monclient: found mon.c 2024-03-29T11:39:57.035 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.033+0000 7fe8a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:57.035 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.033+0000 7fe8a6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:39:57.036 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.033+0000 7fe8a6ffd640 1 -- 172.21.15.72:0/450285980 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b806de70 con 0x7fe8b81422b0 2024-03-29T11:39:57.036 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.033+0000 7fe8a6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:39:57.036 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.033+0000 7fe8a6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:39:57.036 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.033+0000 7fe8a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:57.036 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.034+0000 7fe8bcdf6640 5 monclient: authenticate success, global_id 4412 2024-03-29T11:39:57.036 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.034+0000 7fe8bcdf6640 10 monclient: _renew_subs 2024-03-29T11:39:57.036 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.034+0000 7fe8bcdf6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:39:57.036 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.034+0000 7fe8bcdf6640 1 -- 172.21.15.72:0/450285980 --> v1:172.21.15.72:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8b812e090 con 0x7fe8b81422b0 2024-03-29T11:39:57.036 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.034+0000 7fe8bcdf6640 10 client.4412.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:57.036 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.034+0000 7fe8bcdf6640 10 monclient: _renew_subs 2024-03-29T11:39:57.036 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.034+0000 7fe8bcdf6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:39:57.036 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.034+0000 7fe8bcdf6640 1 -- 172.21.15.72:0/450285980 --> v1:172.21.15.72:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8b8139cf0 con 0x7fe8b81422b0 2024-03-29T11:39:57.036 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.034+0000 7fe8bcdf6640 1 librados: init done 2024-03-29T11:39:57.037 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.035+0000 7fe8bcdf6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:57.037 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.035+0000 7fe8bcdf6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:57.037 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.035+0000 7fe8bcdf6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:39:57.037 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.035+0000 7fe8bcdf6640 1 -- 172.21.15.72:0/450285980 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8b8075f10 con 0x7fe8b81422b0 2024-03-29T11:39:57.038 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.037+0000 7fe8a6ffd640 1 -- 172.21.15.72:0/450285980 <== mon.2 v1:172.21.15.72:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8a8001d60 con 0x7fe8b81422b0 2024-03-29T11:39:57.039 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.037+0000 7fe8a6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:57.039 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.037+0000 7fe8a6ffd640 1 -- 172.21.15.72:0/450285980 <== mon.2 v1:172.21.15.72:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fe8a8004940 con 0x7fe8b81422b0 2024-03-29T11:39:57.039 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.037+0000 7fe8a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:57.039 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.037+0000 7fe8a6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:57.039 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.037+0000 7fe8a6ffd640 10 monclient: dump: 2024-03-29T11:39:57.039 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:39:57.039 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:39:57.039 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:57.039 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:39:57.039 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:57.039 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:39:57.039 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:39:57.039 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:39:57.039 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:39:57.039 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:39:57.040 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.037+0000 7fe8a7fff640 4 set_mon_vals no callback set 2024-03-29T11:39:57.041 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.039+0000 7fe8a6ffd640 1 -- 172.21.15.72:0/450285980 <== mon.2 v1:172.21.15.72:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7fe8a801a910 con 0x7fe8b81422b0 2024-03-29T11:39:57.043 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.041+0000 7fe8a6ffd640 1 -- 172.21.15.72:0/450285980 <== mon.2 v1:172.21.15.72:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7fe8a80848b0 con 0x7fe8b81422b0 2024-03-29T11:39:57.043 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.041+0000 7fe8a6ffd640 10 client.4412.objecter ms_dispatch 0x7fe8b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:57.043 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.041+0000 7fe8a6ffd640 3 client.4412.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:57.043 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.041+0000 7fe8a6ffd640 3 client.4412.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:57.043 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.041+0000 7fe8a6ffd640 20 client.4412.objecter dump_active .. 0 homeless 2024-03-29T11:39:57.043 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.042+0000 7fe8b659c640 10 monclient: get_auth_request con 0x7fe89806dc70 auth_method 0 2024-03-29T11:39:57.044 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.043+0000 7fe8a6ffd640 10 client.4412.objecter ms_handle_connect 0x7fe89806dc70 2024-03-29T11:39:57.047 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.046+0000 7fe8a6ffd640 1 -- 172.21.15.72:0/450285980 <== mon.2 v1:172.21.15.72:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe8a80501f0 con 0x7fe8b81422b0 2024-03-29T11:39:57.048 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.046+0000 7fe8a6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:57.048 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.046+0000 7fe8a6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:57.372 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.370+0000 7fe8bcdf6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:57.372 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.370+0000 7fe8bcdf6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:57.372 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.370+0000 7fe8bcdf6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:39:57.372 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.370+0000 7fe8bcdf6640 1 -- 172.21.15.72:0/450285980 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8b8071f70 con 0x7fe8b81422b0 2024-03-29T11:39:57.373 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.371+0000 7fe8a6ffd640 1 -- 172.21.15.72:0/450285980 <== mon.2 v1:172.21.15.72:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7fe8a8052120 con 0x7fe8b81422b0 2024-03-29T11:39:57.373 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.372+0000 7fe8a6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:57.373 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.372+0000 7fe8a6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:57.376 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:39:57.376 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:57.378 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.377+0000 7fe8bcdf6640 10 librados: watch_flush enter 2024-03-29T11:39:57.378 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.377+0000 7fe8bcdf6640 10 librados: watch_flush exit 2024-03-29T11:39:57.378 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.377+0000 7fe8bcdf6640 20 client.4412.objecter shutdown clearing up homeless session... 2024-03-29T11:39:57.379 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.377+0000 7fe8bcdf6640 10 client.4412.objecter successfully canceled tick 2024-03-29T11:39:57.379 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.378+0000 7fe8bcdf6640 1 -- 172.21.15.72:0/450285980 >> v1:172.21.15.72:6816/1578866102 conn(0x7fe89806dc70 legacy=0x7fe898070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:57.379 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.378+0000 7fe8bcdf6640 10 monclient: shutdown 2024-03-29T11:39:57.379 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.378+0000 7fe8bcdf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:57.379 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.378+0000 7fe8bcdf6640 1 -- 172.21.15.72:0/450285980 >> v1:172.21.15.72:6790/0 conn(0x7fe8b81422b0 legacy=0x7fe8b814d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:57.379 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.378+0000 7fe8bcdf6640 1 -- 172.21.15.72:0/450285980 shutdown_connections 2024-03-29T11:39:57.380 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.378+0000 7fe8bcdf6640 1 -- 172.21.15.72:0/450285980 >> 172.21.15.72:0/450285980 conn(0x7fe8b8066a00 msgr2=0x7fe8b812ac50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:57.380 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.378+0000 7fe8bcdf6640 1 -- 172.21.15.72:0/450285980 shutdown_connections 2024-03-29T11:39:57.380 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.378+0000 7fe8bcdf6640 1 -- 172.21.15.72:0/450285980 wait complete. 2024-03-29T11:39:57.380 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:39:57.379+0000 7fe8bcdf6640 1 librados: shutdown 2024-03-29T11:39:57.399 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:04.206 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:04.361 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.358+0000 7f33e9aae640 10 monclient: get_monmap_and_config 2024-03-29T11:40:04.361 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.359+0000 7f33e9aae640 10 monclient: build_initial_monmap 2024-03-29T11:40:04.361 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.359+0000 7f33e9aae640 10 monclient: monmap: 2024-03-29T11:40:04.361 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:40:04.361 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:40:04.361 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:40:04.359990+0000 2024-03-29T11:40:04.361 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:40:04.359990+0000 2024-03-29T11:40:04.361 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:04.361 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:04.361 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:40:04.361 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:40:04.361 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:40:04.361 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:04.364 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.361+0000 7f33e9aae640 1 Processor -- start 2024-03-29T11:40:04.365 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.362+0000 7f33e9aae640 1 -- start start 2024-03-29T11:40:04.365 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.362+0000 7f33e9aae640 10 monclient: init 2024-03-29T11:40:04.365 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.363+0000 7f33e9aae640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:04.365 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.364+0000 7f33e9aae640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:40:04.366 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.364+0000 7f33e9aae640 10 monclient(hunting): picked mon.noname-c con 0x7f33e40a9750 addr v1:172.21.15.89:6789/0 2024-03-29T11:40:04.366 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.364+0000 7f33e9aae640 10 monclient(hunting): picked mon.noname-b con 0x7f33e40af2e0 addr v1:172.21.15.72:6790/0 2024-03-29T11:40:04.366 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.364+0000 7f33e9aae640 10 monclient(hunting): picked mon.noname-a con 0x7f33e4141ef0 addr v1:172.21.15.72:6789/0 2024-03-29T11:40:04.366 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.364+0000 7f33e9aae640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33e40a9b30 con 0x7f33e4141ef0 2024-03-29T11:40:04.366 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.364+0000 7f33e9aae640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33e40ac4f0 con 0x7f33e40a9750 2024-03-29T11:40:04.367 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.364+0000 7f33e9aae640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33e40af770 con 0x7f33e40af2e0 2024-03-29T11:40:04.367 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.364+0000 7f33e9aae640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:04.367 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.364+0000 7f33e9aae640 10 monclient(hunting): authenticate will time out at 1163.652540s 2024-03-29T11:40:04.369 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.367+0000 7f33e2ffd640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f33e40af2e0 0x7f33e41417e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:40151/0 (socket says 172.21.15.72:40151) 2024-03-29T11:40:04.369 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.367+0000 7f33e2ffd640 1 -- 172.21.15.72:0/2575732552 learned_addr learned my addr 172.21.15.72:0/2575732552 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:40:04.375 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.368+0000 7f33e27fc640 1 -- 172.21.15.72:0/2575732552 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2284787918 0 0) 0x7f33e40af770 con 0x7f33e40af2e0 2024-03-29T11:40:04.375 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.368+0000 7f33e27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:40:04.375 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.368+0000 7f33e27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:04.376 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.368+0000 7f33e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:04.376 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.368+0000 7f33e27fc640 10 monclient(hunting): my global_id is 4415 2024-03-29T11:40:04.376 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.368+0000 7f33e27fc640 1 -- 172.21.15.72:0/2575732552 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33cc003520 con 0x7f33e40af2e0 2024-03-29T11:40:04.376 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.371+0000 7f33e27fc640 1 -- 172.21.15.72:0/2575732552 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3319872603 0 0) 0x7f33e40ac4f0 con 0x7f33e40a9750 2024-03-29T11:40:04.376 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.371+0000 7f33e27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:40:04.376 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.371+0000 7f33e27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:04.376 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.371+0000 7f33e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:04.376 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.371+0000 7f33e27fc640 10 monclient(hunting): my global_id is 4378 2024-03-29T11:40:04.376 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.371+0000 7f33e27fc640 1 -- 172.21.15.72:0/2575732552 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33e40af770 con 0x7f33e40a9750 2024-03-29T11:40:04.376 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.371+0000 7f33e27fc640 1 -- 172.21.15.72:0/2575732552 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1013826746 0 0) 0x7f33cc003520 con 0x7f33e40af2e0 2024-03-29T11:40:04.376 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.371+0000 7f33e27fc640 1 -- 172.21.15.72:0/2575732552 >> v1:172.21.15.89:6789/0 conn(0x7f33e40a9750 legacy=0x7f33e413b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:04.376 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.371+0000 7f33e27fc640 1 -- 172.21.15.72:0/2575732552 >> v1:172.21.15.72:6789/0 conn(0x7f33e4141ef0 legacy=0x7f33e41498e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:04.376 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.371+0000 7f33e27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:40:04.376 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.371+0000 7f33e27fc640 1 monclient: found mon.noname-b 2024-03-29T11:40:04.376 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.371+0000 7f33e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:04.377 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.371+0000 7f33e27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.72:6790/0 2024-03-29T11:40:04.377 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.371+0000 7f33e27fc640 1 -- 172.21.15.72:0/2575732552 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33e40b00c0 con 0x7f33e40af2e0 2024-03-29T11:40:04.377 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.371+0000 7f33e27fc640 10 monclient: _finish_auth 0 2024-03-29T11:40:04.377 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.371+0000 7f33e27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:40:04.377 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.371+0000 7f33e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:04.377 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.371+0000 7f33e27fc640 1 -- 172.21.15.72:0/2575732552 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f33d8001690 con 0x7f33e40af2e0 2024-03-29T11:40:04.377 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.371+0000 7f33e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:04.377 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.371+0000 7f33e27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:04.377 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.371+0000 7f33e27fc640 10 monclient: dump: 2024-03-29T11:40:04.377 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:40:04.377 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:40:04.377 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:04.377 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:04.377 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:04.377 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:04.378 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:40:04.378 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:40:04.378 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:40:04.378 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:04.378 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.372+0000 7f33e9aae640 5 monclient: authenticate success, global_id 4415 2024-03-29T11:40:04.378 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.372+0000 7f33e9aae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:04.378 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.372+0000 7f33e27fc640 1 -- 172.21.15.72:0/2575732552 <== mon.2 v1:172.21.15.72:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f33d80017f0 con 0x7f33e40af2e0 2024-03-29T11:40:04.378 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.372+0000 7f33e27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:04.378 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.372+0000 7f33e9aae640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:04.378 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.372+0000 7f33e9aae640 4 set_mon_vals no callback set 2024-03-29T11:40:04.378 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.373+0000 7f33e27fc640 1 -- 172.21.15.72:0/2575732552 <== mon.2 v1:172.21.15.72:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f33d8001b80 con 0x7f33e40af2e0 2024-03-29T11:40:04.378 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.373+0000 7f33e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:04.378 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.373+0000 7f33e27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:04.378 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.373+0000 7f33e27fc640 10 monclient: dump: 2024-03-29T11:40:04.378 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:40:04.378 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:40:04.379 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:04.379 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:04.379 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:04.379 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:04.379 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:40:04.379 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:40:04.379 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:40:04.379 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:04.379 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.373+0000 7f33e9aae640 10 monclient: shutdown 2024-03-29T11:40:04.379 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.373+0000 7f33e9aae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:04.379 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.373+0000 7f33e9aae640 1 -- 172.21.15.72:0/2575732552 >> v1:172.21.15.72:6790/0 conn(0x7f33e40af2e0 legacy=0x7f33e41417e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:04.379 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.373+0000 7f33e9aae640 1 -- 172.21.15.72:0/2575732552 shutdown_connections 2024-03-29T11:40:04.379 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.373+0000 7f33e9aae640 1 -- 172.21.15.72:0/2575732552 >> 172.21.15.72:0/2575732552 conn(0x7f33e40b3a00 msgr2=0x7f33e40b3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:04.380 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.376+0000 7f33e9aae640 1 -- 172.21.15.72:0/2575732552 shutdown_connections 2024-03-29T11:40:04.380 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.378+0000 7f33e9aae640 1 -- 172.21.15.72:0/2575732552 wait complete. 2024-03-29T11:40:04.381 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.379+0000 7f33e9aae640 10 monclient: build_initial_monmap 2024-03-29T11:40:04.381 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.379+0000 7f33e9aae640 10 monclient: monmap: 2024-03-29T11:40:04.381 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:40:04.381 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:04.381 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:40:04.382 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:40:04.382 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:04.382 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:04.382 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:40:04.382 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:40:04.382 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:40:04.382 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:04.382 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.380+0000 7f33e9aae640 1 librados: starting msgr at 2024-03-29T11:40:04.382 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.380+0000 7f33e9aae640 1 librados: starting objecter 2024-03-29T11:40:04.382 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.380+0000 7f33e9aae640 1 Processor -- start 2024-03-29T11:40:04.383 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.380+0000 7f33e9aae640 1 -- start start 2024-03-29T11:40:04.383 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.380+0000 7f33e9aae640 1 librados: setting wanted keys 2024-03-29T11:40:04.383 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.380+0000 7f33e9aae640 1 librados: calling monclient init 2024-03-29T11:40:04.383 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.380+0000 7f33e9aae640 10 monclient: init 2024-03-29T11:40:04.383 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.381+0000 7f33e9aae640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:04.383 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.382+0000 7f33e9aae640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:40:04.383 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.382+0000 7f33e9aae640 10 monclient(hunting): picked mon.noname-a con 0x7f33e40a9750 addr v1:172.21.15.72:6789/0 2024-03-29T11:40:04.387 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.382+0000 7f33e9aae640 10 monclient(hunting): picked mon.noname-c con 0x7f33e4141ef0 addr v1:172.21.15.72:6790/0 2024-03-29T11:40:04.387 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.385+0000 7f33e9aae640 10 monclient(hunting): picked mon.noname-b con 0x7f33e40b22b0 addr v1:172.21.15.89:6789/0 2024-03-29T11:40:04.387 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.385+0000 7f33e9aae640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33e413e560 con 0x7f33e40a9750 2024-03-29T11:40:04.387 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.385+0000 7f33e9aae640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33e413d3b0 con 0x7f33e40b22b0 2024-03-29T11:40:04.387 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.385+0000 7f33e2ffd640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f33e4141ef0 0x7f33e41d6200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:40231/0 (socket says 172.21.15.72:40231) 2024-03-29T11:40:04.387 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.385+0000 7f33e2ffd640 1 -- 172.21.15.72:0/1091169396 learned_addr learned my addr 172.21.15.72:0/1091169396 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:40:04.387 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.386+0000 7f33e9aae640 1 -- 172.21.15.72:0/1091169396 --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33e413d540 con 0x7f33e4141ef0 2024-03-29T11:40:04.387 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.386+0000 7f33e9aae640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:04.389 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.386+0000 7f33e9aae640 10 monclient(hunting): authenticate will time out at 1163.674239s 2024-03-29T11:40:04.389 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.386+0000 7f33cbfff640 10 client.?.objecter ms_handle_connect 0x7f33e4141ef0 2024-03-29T11:40:04.389 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.386+0000 7f33cbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:04.389 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.386+0000 7f33cbfff640 1 -- 172.21.15.72:0/1091169396 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2969260730 0 0) 0x7f33e413d540 con 0x7f33e4141ef0 2024-03-29T11:40:04.389 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.386+0000 7f33cbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:40:04.389 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.386+0000 7f33cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:04.389 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.386+0000 7f33cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:04.389 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.386+0000 7f33cbfff640 10 monclient(hunting): my global_id is 4418 2024-03-29T11:40:04.390 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.386+0000 7f33cbfff640 1 -- 172.21.15.72:0/1091169396 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33d0004eb0 con 0x7f33e4141ef0 2024-03-29T11:40:04.390 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.387+0000 7f33cbfff640 1 -- 172.21.15.72:0/1091169396 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 336767785 0 0) 0x7f33d0004eb0 con 0x7f33e4141ef0 2024-03-29T11:40:04.390 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.387+0000 7f33cbfff640 1 -- 172.21.15.72:0/1091169396 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33e413d540 con 0x7f33e4141ef0 2024-03-29T11:40:04.390 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.387+0000 7f33cbfff640 1 -- 172.21.15.72:0/1091169396 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f33d8001550 con 0x7f33e4141ef0 2024-03-29T11:40:04.390 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.387+0000 7f33cbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:04.390 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.387+0000 7f33cbfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:04.390 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.387+0000 7f33cbfff640 10 monclient(hunting): dump: 2024-03-29T11:40:04.390 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:40:04.390 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:40:04.390 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:04.390 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:04.390 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:04.390 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:04.390 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:40:04.390 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:40:04.390 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:40:04.391 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:04.391 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.387+0000 7f33cbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:04.395 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.389+0000 7f33cbfff640 1 -- 172.21.15.72:0/1091169396 <== mon.2 v1:172.21.15.72:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 364426058 0 0) 0x7f33e413d540 con 0x7f33e4141ef0 2024-03-29T11:40:04.395 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.390+0000 7f33cbfff640 1 -- 172.21.15.72:0/1091169396 >> v1:172.21.15.89:6789/0 conn(0x7f33e40b22b0 legacy=0x7f33e41d6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:04.395 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.390+0000 7f33cbfff640 1 -- 172.21.15.72:0/1091169396 >> v1:172.21.15.72:6789/0 conn(0x7f33e40a9750 legacy=0x7f33e413dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:04.395 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.390+0000 7f33cbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:40:04.395 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.390+0000 7f33cbfff640 1 monclient: found mon.c 2024-03-29T11:40:04.395 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.390+0000 7f33cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:04.395 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.390+0000 7f33cbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:40:04.395 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.390+0000 7f33cbfff640 1 -- 172.21.15.72:0/1091169396 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33e41e0740 con 0x7f33e4141ef0 2024-03-29T11:40:04.395 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.390+0000 7f33cbfff640 10 monclient: _finish_auth 0 2024-03-29T11:40:04.395 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.390+0000 7f33cbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:40:04.395 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.390+0000 7f33cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:04.395 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.390+0000 7f33cbfff640 10 client.?.objecter ms_handle_connect 0x7f33e40a9750 2024-03-29T11:40:04.395 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.390+0000 7f33cbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:04.395 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.392+0000 7f33e9aae640 5 monclient: authenticate success, global_id 4418 2024-03-29T11:40:04.396 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.392+0000 7f33e9aae640 10 monclient: _renew_subs 2024-03-29T11:40:04.396 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.392+0000 7f33e9aae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:40:04.396 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.392+0000 7f33e9aae640 1 -- 172.21.15.72:0/1091169396 --> v1:172.21.15.72:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33e41de720 con 0x7f33e4141ef0 2024-03-29T11:40:04.396 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.392+0000 7f33e9aae640 10 client.4418.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:04.396 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.392+0000 7f33e9aae640 10 monclient: _renew_subs 2024-03-29T11:40:04.396 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.392+0000 7f33e9aae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:40:04.396 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.392+0000 7f33e9aae640 1 -- 172.21.15.72:0/1091169396 --> v1:172.21.15.72:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f33e41dec00 con 0x7f33e4141ef0 2024-03-29T11:40:04.396 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.392+0000 7f33e9aae640 1 librados: init done 2024-03-29T11:40:04.396 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.394+0000 7f33e9aae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:04.397 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.394+0000 7f33e9aae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:04.397 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.394+0000 7f33e9aae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:40:04.397 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.394+0000 7f33e9aae640 1 -- 172.21.15.72:0/1091169396 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33e413e8d0 con 0x7f33e4141ef0 2024-03-29T11:40:04.398 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.397+0000 7f33cbfff640 1 -- 172.21.15.72:0/1091169396 <== mon.2 v1:172.21.15.72:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f33d8004a90 con 0x7f33e4141ef0 2024-03-29T11:40:04.398 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.397+0000 7f33cbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:04.398 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.397+0000 7f33cbfff640 1 -- 172.21.15.72:0/1091169396 <== mon.2 v1:172.21.15.72:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f33d8004e20 con 0x7f33e4141ef0 2024-03-29T11:40:04.398 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.397+0000 7f33cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:04.398 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.397+0000 7f33cbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:04.398 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.397+0000 7f33cbfff640 10 monclient: dump: 2024-03-29T11:40:04.399 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:40:04.399 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:40:04.399 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:04.399 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:04.399 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:04.399 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:04.399 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:40:04.399 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:40:04.399 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:40:04.399 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:04.399 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.397+0000 7f33e1ffb640 4 set_mon_vals no callback set 2024-03-29T11:40:04.400 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.398+0000 7f33cbfff640 1 -- 172.21.15.72:0/1091169396 <== mon.2 v1:172.21.15.72:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f33d8005040 con 0x7f33e4141ef0 2024-03-29T11:40:04.400 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.399+0000 7f33cbfff640 1 -- 172.21.15.72:0/1091169396 <== mon.2 v1:172.21.15.72:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f33d8084c20 con 0x7f33e4141ef0 2024-03-29T11:40:04.400 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.399+0000 7f33cbfff640 10 client.4418.objecter ms_dispatch 0x7f33e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:04.400 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.399+0000 7f33cbfff640 3 client.4418.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:04.401 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.399+0000 7f33cbfff640 3 client.4418.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:04.401 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.399+0000 7f33cbfff640 20 client.4418.objecter dump_active .. 0 homeless 2024-03-29T11:40:04.401 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.399+0000 7f33e37fe640 10 monclient: get_auth_request con 0x7f33d006dca0 auth_method 0 2024-03-29T11:40:04.401 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.400+0000 7f33cbfff640 10 client.4418.objecter ms_handle_connect 0x7f33d006dca0 2024-03-29T11:40:04.407 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.406+0000 7f33cbfff640 1 -- 172.21.15.72:0/1091169396 <== mon.2 v1:172.21.15.72:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f33d8050500 con 0x7f33e4141ef0 2024-03-29T11:40:04.407 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.406+0000 7f33cbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:04.408 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.406+0000 7f33cbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:04.738 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.736+0000 7f33e9aae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:04.739 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.736+0000 7f33e9aae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:04.739 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.736+0000 7f33e9aae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:40:04.739 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.736+0000 7f33e9aae640 1 -- 172.21.15.72:0/1091169396 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f33e404f730 con 0x7f33e4141ef0 2024-03-29T11:40:04.741 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.739+0000 7f33cbfff640 1 -- 172.21.15.72:0/1091169396 <== mon.2 v1:172.21.15.72:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f33d8052430 con 0x7f33e4141ef0 2024-03-29T11:40:04.741 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.739+0000 7f33cbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:04.741 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.739+0000 7f33cbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:04.741 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:40:04.741 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:04.745 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.743+0000 7f33e9aae640 10 librados: watch_flush enter 2024-03-29T11:40:04.745 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.743+0000 7f33e9aae640 10 librados: watch_flush exit 2024-03-29T11:40:04.745 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.743+0000 7f33e9aae640 20 client.4418.objecter shutdown clearing up homeless session... 2024-03-29T11:40:04.745 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.743+0000 7f33e9aae640 10 client.4418.objecter successfully canceled tick 2024-03-29T11:40:04.745 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.743+0000 7f33e9aae640 1 -- 172.21.15.72:0/1091169396 >> v1:172.21.15.72:6816/1578866102 conn(0x7f33d006dca0 legacy=0x7f33d0070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:04.745 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.743+0000 7f33e9aae640 10 monclient: shutdown 2024-03-29T11:40:04.745 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.743+0000 7f33e9aae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:04.745 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.743+0000 7f33e9aae640 1 -- 172.21.15.72:0/1091169396 >> v1:172.21.15.72:6790/0 conn(0x7f33e4141ef0 legacy=0x7f33e41d6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:04.745 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.743+0000 7f33e9aae640 1 -- 172.21.15.72:0/1091169396 shutdown_connections 2024-03-29T11:40:04.745 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.743+0000 7f33e9aae640 1 -- 172.21.15.72:0/1091169396 >> 172.21.15.72:0/1091169396 conn(0x7f33e40b3a00 msgr2=0x7f33e413a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:04.746 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.744+0000 7f33e9aae640 1 -- 172.21.15.72:0/1091169396 shutdown_connections 2024-03-29T11:40:04.746 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.744+0000 7f33e9aae640 1 -- 172.21.15.72:0/1091169396 wait complete. 2024-03-29T11:40:04.746 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:04.744+0000 7f33e9aae640 1 librados: shutdown 2024-03-29T11:40:04.763 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:11.570 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:11.720 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.718+0000 7feeefe4e640 10 monclient: get_monmap_and_config 2024-03-29T11:40:11.720 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.719+0000 7feeefe4e640 10 monclient: build_initial_monmap 2024-03-29T11:40:11.720 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.719+0000 7feeefe4e640 10 monclient: monmap: 2024-03-29T11:40:11.721 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:40:11.721 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:40:11.721 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:40:11.719992+0000 2024-03-29T11:40:11.721 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:40:11.719992+0000 2024-03-29T11:40:11.721 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:11.721 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:11.721 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:40:11.721 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:40:11.721 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:40:11.721 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:11.722 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.719+0000 7feeefe4e640 1 Processor -- start 2024-03-29T11:40:11.722 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.720+0000 7feeefe4e640 1 -- start start 2024-03-29T11:40:11.722 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.720+0000 7feeefe4e640 10 monclient: init 2024-03-29T11:40:11.722 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.721+0000 7feeefe4e640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:11.722 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.721+0000 7feeefe4e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:40:11.725 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.721+0000 7feeefe4e640 10 monclient(hunting): picked mon.noname-a con 0x7feee81422b0 addr v1:172.21.15.72:6789/0 2024-03-29T11:40:11.725 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.724+0000 7feeefe4e640 10 monclient(hunting): picked mon.noname-b con 0x7feee8142690 addr v1:172.21.15.72:6790/0 2024-03-29T11:40:11.729 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.727+0000 7feeedbea640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7feee81422b0 0x7feee813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:44075/0 (socket says 172.21.15.72:44075) 2024-03-29T11:40:11.729 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.727+0000 7feeefe4e640 10 monclient(hunting): picked mon.noname-c con 0x7feee80b17d0 addr v1:172.21.15.89:6789/0 2024-03-29T11:40:11.729 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.727+0000 7feeefe4e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee8142c80 con 0x7feee81422b0 2024-03-29T11:40:11.729 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.727+0000 7feeedbea640 1 -- 172.21.15.72:0/1729379715 learned_addr learned my addr 172.21.15.72:0/1729379715 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:40:11.730 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.728+0000 7feeefe4e640 1 -- 172.21.15.72:0/1729379715 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee80aae40 con 0x7feee80b17d0 2024-03-29T11:40:11.730 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.728+0000 7feeefe4e640 1 -- 172.21.15.72:0/1729379715 --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee813fbd0 con 0x7feee8142690 2024-03-29T11:40:11.730 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.728+0000 7feeefe4e640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:11.730 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.728+0000 7feeefe4e640 10 monclient(hunting): authenticate will time out at 1171.016313s 2024-03-29T11:40:11.731 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.729+0000 7feeecbe8640 1 -- 172.21.15.72:0/1729379715 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3133606798 0 0) 0x7feee813fbd0 con 0x7feee8142690 2024-03-29T11:40:11.731 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.729+0000 7feeecbe8640 10 monclient(hunting): _negotiate 2024-03-29T11:40:11.731 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.729+0000 7feeecbe8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:11.731 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.729+0000 7feeecbe8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:11.731 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.729+0000 7feeecbe8640 10 monclient(hunting): my global_id is 4421 2024-03-29T11:40:11.732 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.729+0000 7feeecbe8640 1 -- 172.21.15.72:0/1729379715 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feed4003520 con 0x7feee8142690 2024-03-29T11:40:11.732 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.730+0000 7feeecbe8640 1 -- 172.21.15.72:0/1729379715 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2268472423 0 0) 0x7feee80aae40 con 0x7feee80b17d0 2024-03-29T11:40:11.732 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.730+0000 7feeecbe8640 10 monclient(hunting): _negotiate 2024-03-29T11:40:11.732 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.730+0000 7feeecbe8640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:11.732 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.730+0000 7feeecbe8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:11.732 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.730+0000 7feeecbe8640 10 monclient(hunting): my global_id is 4381 2024-03-29T11:40:11.732 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.730+0000 7feeecbe8640 1 -- 172.21.15.72:0/1729379715 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feee813fbd0 con 0x7feee80b17d0 2024-03-29T11:40:11.733 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.730+0000 7feeecbe8640 1 -- 172.21.15.72:0/1729379715 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 785283538 0 0) 0x7feed4003520 con 0x7feee8142690 2024-03-29T11:40:11.733 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.730+0000 7feeecbe8640 1 -- 172.21.15.72:0/1729379715 >> v1:172.21.15.89:6789/0 conn(0x7feee80b17d0 legacy=0x7feee80b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:11.733 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.730+0000 7feeecbe8640 1 -- 172.21.15.72:0/1729379715 >> v1:172.21.15.72:6789/0 conn(0x7feee81422b0 legacy=0x7feee813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:11.733 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.730+0000 7feeecbe8640 10 monclient: _finish_hunting 0 2024-03-29T11:40:11.733 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.730+0000 7feeecbe8640 1 monclient: found mon.noname-b 2024-03-29T11:40:11.734 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.730+0000 7feeecbe8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:11.734 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.730+0000 7feeecbe8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.72:6790/0 2024-03-29T11:40:11.734 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.730+0000 7feeecbe8640 1 -- 172.21.15.72:0/1729379715 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee8151940 con 0x7feee8142690 2024-03-29T11:40:11.734 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.730+0000 7feeecbe8640 10 monclient: _finish_auth 0 2024-03-29T11:40:11.734 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.730+0000 7feeecbe8640 10 monclient: _check_auth_tickets 2024-03-29T11:40:11.734 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.730+0000 7feeecbe8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:11.734 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.730+0000 7feeecbe8640 1 -- 172.21.15.72:0/1729379715 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7feee4001690 con 0x7feee8142690 2024-03-29T11:40:11.734 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.730+0000 7feeecbe8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:11.734 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.730+0000 7feeecbe8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:11.734 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.730+0000 7feeecbe8640 10 monclient: dump: 2024-03-29T11:40:11.734 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:40:11.734 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:40:11.734 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:11.735 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:11.735 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:11.735 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:11.735 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:40:11.735 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:40:11.735 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:40:11.735 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:11.735 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.731+0000 7feeecbe8640 1 -- 172.21.15.72:0/1729379715 <== mon.2 v1:172.21.15.72:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feee40017f0 con 0x7feee8142690 2024-03-29T11:40:11.735 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.731+0000 7feeefe4e640 5 monclient: authenticate success, global_id 4421 2024-03-29T11:40:11.735 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.731+0000 7feeecbe8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:11.735 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.731+0000 7feeecbe8640 1 -- 172.21.15.72:0/1729379715 <== mon.2 v1:172.21.15.72:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7feee4001b80 con 0x7feee8142690 2024-03-29T11:40:11.735 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.731+0000 7feeecbe8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:11.735 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.731+0000 7feeecbe8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:11.735 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.731+0000 7feeecbe8640 10 monclient: dump: 2024-03-29T11:40:11.735 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:40:11.735 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:40:11.736 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:11.736 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:11.736 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:11.736 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:11.736 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:40:11.736 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:40:11.736 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:40:11.736 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:11.736 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.731+0000 7feeefe4e640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:11.736 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.731+0000 7feeefe4e640 4 set_mon_vals no callback set 2024-03-29T11:40:11.736 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.732+0000 7feeefe4e640 10 monclient: shutdown 2024-03-29T11:40:11.736 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.732+0000 7feeefe4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:11.736 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.732+0000 7feeefe4e640 1 -- 172.21.15.72:0/1729379715 >> v1:172.21.15.72:6790/0 conn(0x7feee8142690 legacy=0x7feee80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:11.737 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.735+0000 7feeefe4e640 1 -- 172.21.15.72:0/1729379715 shutdown_connections 2024-03-29T11:40:11.737 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.735+0000 7feeefe4e640 1 -- 172.21.15.72:0/1729379715 >> 172.21.15.72:0/1729379715 conn(0x7feee8066a00 msgr2=0x7feee8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:11.737 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.735+0000 7feeefe4e640 1 -- 172.21.15.72:0/1729379715 shutdown_connections 2024-03-29T11:40:11.737 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.735+0000 7feeefe4e640 1 -- 172.21.15.72:0/1729379715 wait complete. 2024-03-29T11:40:11.738 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.736+0000 7feeefe4e640 10 monclient: build_initial_monmap 2024-03-29T11:40:11.738 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.736+0000 7feeefe4e640 10 monclient: monmap: 2024-03-29T11:40:11.738 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:40:11.738 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:11.738 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:40:11.738 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:40:11.738 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:11.738 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:11.738 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:40:11.738 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:40:11.738 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:40:11.739 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:11.740 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.737+0000 7feeefe4e640 1 librados: starting msgr at 2024-03-29T11:40:11.740 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.737+0000 7feeefe4e640 1 librados: starting objecter 2024-03-29T11:40:11.740 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.737+0000 7feeefe4e640 1 Processor -- start 2024-03-29T11:40:11.740 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.737+0000 7feeefe4e640 1 -- start start 2024-03-29T11:40:11.740 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.737+0000 7feeefe4e640 1 librados: setting wanted keys 2024-03-29T11:40:11.740 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.737+0000 7feeefe4e640 1 librados: calling monclient init 2024-03-29T11:40:11.740 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.737+0000 7feeefe4e640 10 monclient: init 2024-03-29T11:40:11.740 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.738+0000 7feeefe4e640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:11.740 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.738+0000 7feeefe4e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:40:11.740 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.738+0000 7feeefe4e640 10 monclient(hunting): picked mon.noname-b con 0x7feee80b17d0 addr v1:172.21.15.89:6789/0 2024-03-29T11:40:11.743 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.742+0000 7feeed3e9640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7feee81422b0 0x7feee81daa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:47733/0 (socket says 172.21.15.72:47733) 2024-03-29T11:40:11.743 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.742+0000 7feeed3e9640 1 -- 172.21.15.72:0/893460370 learned_addr learned my addr 172.21.15.72:0/893460370 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:40:11.743 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.742+0000 7feeefe4e640 10 monclient(hunting): picked mon.noname-c con 0x7feee81422b0 addr v1:172.21.15.72:6790/0 2024-03-29T11:40:11.748 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.742+0000 7feede7fc640 10 client.?.objecter ms_handle_connect 0x7feee81422b0 2024-03-29T11:40:11.748 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.746+0000 7feede7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:11.748 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.745+0000 7feeefe4e640 10 monclient(hunting): picked mon.noname-a con 0x7feee8142690 addr v1:172.21.15.72:6789/0 2024-03-29T11:40:11.748 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.746+0000 7feede7fc640 10 client.?.objecter ms_handle_connect 0x7feee80b17d0 2024-03-29T11:40:11.748 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.746+0000 7feeefe4e640 1 -- 172.21.15.72:0/893460370 --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee80af5f0 con 0x7feee8142690 2024-03-29T11:40:11.748 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.746+0000 7feede7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:11.748 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.746+0000 7feede7fc640 10 client.?.objecter ms_handle_connect 0x7feee8142690 2024-03-29T11:40:11.748 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.746+0000 7feede7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:11.748 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.746+0000 7feeefe4e640 1 -- 172.21.15.72:0/893460370 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee80b1060 con 0x7feee80b17d0 2024-03-29T11:40:11.749 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.746+0000 7feede7fc640 1 -- 172.21.15.72:0/893460370 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 872823300 0 0) 0x7feee80af5f0 con 0x7feee8142690 2024-03-29T11:40:11.749 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.746+0000 7feeefe4e640 1 -- 172.21.15.72:0/893460370 --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee80b0090 con 0x7feee81422b0 2024-03-29T11:40:11.749 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.746+0000 7feeefe4e640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:11.749 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.746+0000 7feeefe4e640 10 monclient(hunting): authenticate will time out at 1171.034949s 2024-03-29T11:40:11.753 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.747+0000 7feede7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:40:11.753 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.747+0000 7feede7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:11.754 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.747+0000 7feede7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:11.754 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.747+0000 7feede7fc640 10 monclient(hunting): my global_id is 4410 2024-03-29T11:40:11.754 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.747+0000 7feede7fc640 1 -- 172.21.15.72:0/893460370 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feecc004eb0 con 0x7feee8142690 2024-03-29T11:40:11.754 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.747+0000 7feede7fc640 1 -- 172.21.15.72:0/893460370 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2413363764 0 0) 0x7feee80b0090 con 0x7feee81422b0 2024-03-29T11:40:11.754 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.747+0000 7feede7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:40:11.754 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.747+0000 7feede7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:11.754 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.747+0000 7feede7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:11.754 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.747+0000 7feede7fc640 10 monclient(hunting): my global_id is 4424 2024-03-29T11:40:11.754 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.747+0000 7feede7fc640 1 -- 172.21.15.72:0/893460370 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feee80af5f0 con 0x7feee81422b0 2024-03-29T11:40:11.754 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.748+0000 7feede7fc640 1 -- 172.21.15.72:0/893460370 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1449030248 0 0) 0x7feee80af5f0 con 0x7feee81422b0 2024-03-29T11:40:11.754 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.748+0000 7feede7fc640 1 -- 172.21.15.72:0/893460370 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feee80b0090 con 0x7feee81422b0 2024-03-29T11:40:11.754 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.748+0000 7feede7fc640 1 -- 172.21.15.72:0/893460370 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7feee4001bc0 con 0x7feee81422b0 2024-03-29T11:40:11.754 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.748+0000 7feede7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:11.755 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.748+0000 7feede7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:11.755 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.748+0000 7feede7fc640 10 monclient(hunting): dump: 2024-03-29T11:40:11.755 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:40:11.755 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:40:11.755 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:11.755 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:11.755 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:11.755 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:11.755 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:40:11.755 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:40:11.755 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:40:11.755 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:11.755 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.748+0000 7feede7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:11.755 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.748+0000 7feede7fc640 1 -- 172.21.15.72:0/893460370 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4069963129 0 0) 0x7feecc004eb0 con 0x7feee8142690 2024-03-29T11:40:11.755 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.748+0000 7feede7fc640 1 -- 172.21.15.72:0/893460370 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feee80af5f0 con 0x7feee8142690 2024-03-29T11:40:11.756 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.748+0000 7feede7fc640 1 -- 172.21.15.72:0/893460370 <== mon.2 v1:172.21.15.72:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 11430498 0 0) 0x7feee80b0090 con 0x7feee81422b0 2024-03-29T11:40:11.756 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.748+0000 7feede7fc640 1 -- 172.21.15.72:0/893460370 >> v1:172.21.15.89:6789/0 conn(0x7feee80b17d0 legacy=0x7feee80aeee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:11.756 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.748+0000 7feede7fc640 1 -- 172.21.15.72:0/893460370 >> v1:172.21.15.72:6789/0 conn(0x7feee8142690 legacy=0x7feee81db180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:11.756 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.748+0000 7feede7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:40:11.756 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.748+0000 7feede7fc640 1 monclient: found mon.c 2024-03-29T11:40:11.756 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.748+0000 7feede7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:11.756 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.748+0000 7feede7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:40:11.756 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.748+0000 7feede7fc640 1 -- 172.21.15.72:0/893460370 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee80afc60 con 0x7feee81422b0 2024-03-29T11:40:11.756 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.748+0000 7feede7fc640 10 monclient: _finish_auth 0 2024-03-29T11:40:11.756 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.748+0000 7feede7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:40:11.756 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.748+0000 7feede7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:11.756 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.748+0000 7feeefe4e640 5 monclient: authenticate success, global_id 4424 2024-03-29T11:40:11.756 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.748+0000 7feeefe4e640 10 monclient: _renew_subs 2024-03-29T11:40:11.756 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.748+0000 7feeefe4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:40:11.756 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.748+0000 7feeefe4e640 1 -- 172.21.15.72:0/893460370 --> v1:172.21.15.72:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feee814d7a0 con 0x7feee81422b0 2024-03-29T11:40:11.756 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.748+0000 7feeefe4e640 10 client.4424.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:11.757 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.748+0000 7feeefe4e640 10 monclient: _renew_subs 2024-03-29T11:40:11.757 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.748+0000 7feeefe4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:40:11.757 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.748+0000 7feeefe4e640 1 -- 172.21.15.72:0/893460370 --> v1:172.21.15.72:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feee814dcd0 con 0x7feee81422b0 2024-03-29T11:40:11.757 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.748+0000 7feeefe4e640 1 librados: init done 2024-03-29T11:40:11.757 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.749+0000 7feede7fc640 1 -- 172.21.15.72:0/893460370 <== mon.2 v1:172.21.15.72:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feee40038a0 con 0x7feee81422b0 2024-03-29T11:40:11.757 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.749+0000 7feede7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:11.757 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.749+0000 7feede7fc640 1 -- 172.21.15.72:0/893460370 <== mon.2 v1:172.21.15.72:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7feee4003bf0 con 0x7feee81422b0 2024-03-29T11:40:11.757 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.749+0000 7feede7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:11.757 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.749+0000 7feede7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:11.757 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.749+0000 7feede7fc640 10 monclient: dump: 2024-03-29T11:40:11.757 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:40:11.757 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:40:11.757 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:11.757 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:11.757 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:11.758 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:11.758 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:40:11.758 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:40:11.758 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:40:11.758 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:11.758 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.749+0000 7feedf7fe640 4 set_mon_vals no callback set 2024-03-29T11:40:11.758 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.751+0000 7feede7fc640 1 -- 172.21.15.72:0/893460370 <== mon.2 v1:172.21.15.72:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7feee4003d50 con 0x7feee81422b0 2024-03-29T11:40:11.758 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.752+0000 7feecbfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:11.759 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.752+0000 7feecbfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:11.759 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.752+0000 7feecbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:40:11.759 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.752+0000 7feecbfff640 1 -- 172.21.15.72:0/893460370 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feee80b0220 con 0x7feee81422b0 2024-03-29T11:40:11.759 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.754+0000 7feeedbea640 10 monclient: get_auth_request con 0x7feecc0766a0 auth_method 0 2024-03-29T11:40:11.759 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.754+0000 7feede7fc640 1 -- 172.21.15.72:0/893460370 <== mon.2 v1:172.21.15.72:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7feee4084b40 con 0x7feee81422b0 2024-03-29T11:40:11.759 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.754+0000 7feede7fc640 10 client.4424.objecter ms_dispatch 0x7feee8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:11.759 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.754+0000 7feede7fc640 3 client.4424.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:11.759 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.754+0000 7feede7fc640 3 client.4424.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:11.759 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.754+0000 7feede7fc640 20 client.4424.objecter dump_active .. 0 homeless 2024-03-29T11:40:11.759 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.754+0000 7feede7fc640 10 client.4424.objecter ms_handle_connect 0x7feecc0766a0 2024-03-29T11:40:11.762 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.761+0000 7feede7fc640 1 -- 172.21.15.72:0/893460370 <== mon.2 v1:172.21.15.72:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feee4050420 con 0x7feee81422b0 2024-03-29T11:40:11.762 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.761+0000 7feede7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:11.763 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:11.761+0000 7feede7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:12.084 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:12.081+0000 7feecbfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:12.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:12.081+0000 7feecbfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:12.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:12.081+0000 7feecbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:40:12.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:12.081+0000 7feecbfff640 1 -- 172.21.15.72:0/893460370 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feee8075f10 con 0x7feee81422b0 2024-03-29T11:40:12.086 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:40:12.086 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:12.087 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:12.084+0000 7feede7fc640 1 -- 172.21.15.72:0/893460370 <== mon.2 v1:172.21.15.72:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7feee4052350 con 0x7feee81422b0 2024-03-29T11:40:12.087 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:12.084+0000 7feede7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:12.087 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:12.084+0000 7feede7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:12.089 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:12.088+0000 7feecbfff640 10 librados: watch_flush enter 2024-03-29T11:40:12.089 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:12.088+0000 7feecbfff640 10 librados: watch_flush exit 2024-03-29T11:40:12.089 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:12.088+0000 7feecbfff640 20 client.4424.objecter shutdown clearing up homeless session... 2024-03-29T11:40:12.089 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:12.088+0000 7feecbfff640 10 client.4424.objecter successfully canceled tick 2024-03-29T11:40:12.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:12.090+0000 7feecbfff640 1 -- 172.21.15.72:0/893460370 >> v1:172.21.15.72:6816/1578866102 conn(0x7feecc0766a0 legacy=0x7feecc078b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:12.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:12.090+0000 7feecbfff640 10 monclient: shutdown 2024-03-29T11:40:12.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:12.090+0000 7feecbfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:12.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:12.090+0000 7feecbfff640 1 -- 172.21.15.72:0/893460370 >> v1:172.21.15.72:6790/0 conn(0x7feee81422b0 legacy=0x7feee81daa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:12.093 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:12.092+0000 7feecbfff640 1 -- 172.21.15.72:0/893460370 shutdown_connections 2024-03-29T11:40:12.093 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:12.092+0000 7feecbfff640 1 -- 172.21.15.72:0/893460370 >> 172.21.15.72:0/893460370 conn(0x7feee8066a00 msgr2=0x7feee8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:12.095 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:12.094+0000 7feecbfff640 1 -- 172.21.15.72:0/893460370 shutdown_connections 2024-03-29T11:40:12.096 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:12.095+0000 7feecbfff640 1 -- 172.21.15.72:0/893460370 wait complete. 2024-03-29T11:40:12.096 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:12.095+0000 7feecbfff640 1 librados: shutdown 2024-03-29T11:40:12.113 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:18.920 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:19.065 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.063+0000 7f704a940640 10 monclient: get_monmap_and_config 2024-03-29T11:40:19.066 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.063+0000 7f704a940640 10 monclient: build_initial_monmap 2024-03-29T11:40:19.066 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.063+0000 7f704a940640 10 monclient: monmap: 2024-03-29T11:40:19.066 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:40:19.066 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:40:19.066 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:40:19.064914+0000 2024-03-29T11:40:19.066 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:40:19.064914+0000 2024-03-29T11:40:19.066 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:19.066 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:19.066 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:40:19.066 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:40:19.066 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:40:19.066 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:19.067 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.065+0000 7f704a940640 1 Processor -- start 2024-03-29T11:40:19.067 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.065+0000 7f704a940640 1 -- start start 2024-03-29T11:40:19.067 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.065+0000 7f704a940640 10 monclient: init 2024-03-29T11:40:19.068 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.066+0000 7f704a940640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:19.068 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.066+0000 7f704a940640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:40:19.068 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.066+0000 7f704a940640 10 monclient(hunting): picked mon.noname-b con 0x7f70440a9750 addr v1:172.21.15.72:6790/0 2024-03-29T11:40:19.068 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.066+0000 7f704a940640 10 monclient(hunting): picked mon.noname-a con 0x7f704413e1e0 addr v1:172.21.15.72:6789/0 2024-03-29T11:40:19.068 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.066+0000 7f704a940640 10 monclient(hunting): picked mon.noname-c con 0x7f7044071c30 addr v1:172.21.15.89:6789/0 2024-03-29T11:40:19.068 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.066+0000 7f704a940640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70440ab300 con 0x7f704413e1e0 2024-03-29T11:40:19.068 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.066+0000 7f704a940640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70440ac2f0 con 0x7f7044071c30 2024-03-29T11:40:19.069 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.066+0000 7f704a940640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70440ad8d0 con 0x7f70440a9750 2024-03-29T11:40:19.069 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.066+0000 7f704a940640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:19.069 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.066+0000 7f704a940640 10 monclient(hunting): authenticate will time out at 1178.354592s 2024-03-29T11:40:19.069 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.066+0000 7f7048edd640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f7044071c30 0x7f70440ace20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:45138/0 (socket says 172.21.15.72:45138) 2024-03-29T11:40:19.069 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.066+0000 7f7048edd640 1 -- 172.21.15.72:0/707393046 learned_addr learned my addr 172.21.15.72:0/707393046 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:40:19.070 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.067+0000 7f7042ffd640 1 -- 172.21.15.72:0/707393046 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2202300756 0 0) 0x7f70440ab300 con 0x7f704413e1e0 2024-03-29T11:40:19.070 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.067+0000 7f7042ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:19.070 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.067+0000 7f7042ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:19.071 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.067+0000 7f7042ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:19.071 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.067+0000 7f7042ffd640 10 monclient(hunting): my global_id is 4413 2024-03-29T11:40:19.071 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.067+0000 7f7042ffd640 1 -- 172.21.15.72:0/707393046 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f702c005730 con 0x7f704413e1e0 2024-03-29T11:40:19.071 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.067+0000 7f7042ffd640 1 -- 172.21.15.72:0/707393046 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 66670126 0 0) 0x7f70440ad8d0 con 0x7f70440a9750 2024-03-29T11:40:19.071 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.067+0000 7f7042ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:19.071 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.067+0000 7f7042ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:19.071 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.067+0000 7f7042ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:19.071 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.067+0000 7f7042ffd640 10 monclient(hunting): my global_id is 4427 2024-03-29T11:40:19.071 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.068+0000 7f7042ffd640 1 -- 172.21.15.72:0/707393046 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70440ab300 con 0x7f70440a9750 2024-03-29T11:40:19.071 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.068+0000 7f7042ffd640 1 -- 172.21.15.72:0/707393046 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2563704405 0 0) 0x7f702c005730 con 0x7f704413e1e0 2024-03-29T11:40:19.071 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.068+0000 7f7042ffd640 1 -- 172.21.15.72:0/707393046 >> v1:172.21.15.72:6790/0 conn(0x7f70440a9750 legacy=0x7f704413dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:19.071 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.068+0000 7f7042ffd640 1 -- 172.21.15.72:0/707393046 >> v1:172.21.15.89:6789/0 conn(0x7f7044071c30 legacy=0x7f70440ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:40:19.071 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.068+0000 7f7042ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:40:19.071 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.068+0000 7f7042ffd640 1 monclient: found mon.noname-a 2024-03-29T11:40:19.071 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.068+0000 7f7042ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:19.072 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.068+0000 7f7042ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.72:6789/0 2024-03-29T11:40:19.072 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.068+0000 7f7042ffd640 1 -- 172.21.15.72:0/707393046 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70440ada60 con 0x7f704413e1e0 2024-03-29T11:40:19.072 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.068+0000 7f7042ffd640 10 monclient: _finish_auth 0 2024-03-29T11:40:19.072 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.068+0000 7f7042ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:40:19.072 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.068+0000 7f7042ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:19.072 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.068+0000 7f7042ffd640 1 -- 172.21.15.72:0/707393046 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f7038001690 con 0x7f704413e1e0 2024-03-29T11:40:19.072 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.068+0000 7f7042ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:19.072 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.068+0000 7f7042ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:40:19.072 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.068+0000 7f7042ffd640 10 monclient: dump: 2024-03-29T11:40:19.072 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:40:19.072 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:40:19.072 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:19.072 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:19.072 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:19.072 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:19.072 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:40:19.072 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:40:19.073 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:40:19.073 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:19.073 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.068+0000 7f704a940640 5 monclient: authenticate success, global_id 4413 2024-03-29T11:40:19.073 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.068+0000 7f704a940640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:19.073 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.068+0000 7f7042ffd640 1 -- 172.21.15.72:0/707393046 <== mon.0 v1:172.21.15.72:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70380017f0 con 0x7f704413e1e0 2024-03-29T11:40:19.073 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.068+0000 7f7042ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:19.073 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.068+0000 7f7042ffd640 1 -- 172.21.15.72:0/707393046 <== mon.0 v1:172.21.15.72:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f7038001c00 con 0x7f704413e1e0 2024-03-29T11:40:19.073 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.068+0000 7f7042ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:19.073 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.068+0000 7f7042ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:40:19.073 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.068+0000 7f7042ffd640 10 monclient: dump: 2024-03-29T11:40:19.073 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:40:19.073 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:40:19.074 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:19.074 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:19.074 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:19.074 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:19.074 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:40:19.074 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:40:19.074 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:40:19.074 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:19.074 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.068+0000 7f704a940640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:19.074 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.068+0000 7f704a940640 4 set_mon_vals no callback set 2024-03-29T11:40:19.082 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.069+0000 7f704a940640 10 monclient: shutdown 2024-03-29T11:40:19.082 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.069+0000 7f704a940640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:19.082 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.069+0000 7f704a940640 1 -- 172.21.15.72:0/707393046 >> v1:172.21.15.72:6789/0 conn(0x7f704413e1e0 legacy=0x7f70440ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:19.082 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.069+0000 7f704a940640 1 -- 172.21.15.72:0/707393046 shutdown_connections 2024-03-29T11:40:19.082 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.069+0000 7f704a940640 1 -- 172.21.15.72:0/707393046 >> 172.21.15.72:0/707393046 conn(0x7f704406b660 msgr2=0x7f704406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:19.082 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.069+0000 7f704a940640 1 -- 172.21.15.72:0/707393046 shutdown_connections 2024-03-29T11:40:19.082 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.069+0000 7f704a940640 1 -- 172.21.15.72:0/707393046 wait complete. 2024-03-29T11:40:19.083 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.070+0000 7f704a940640 10 monclient: build_initial_monmap 2024-03-29T11:40:19.083 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.070+0000 7f704a940640 10 monclient: monmap: 2024-03-29T11:40:19.083 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:40:19.083 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:19.083 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:40:19.083 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:40:19.083 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:19.083 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:19.083 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:40:19.083 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:40:19.083 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:40:19.083 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:19.083 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.071+0000 7f704a940640 1 librados: starting msgr at 2024-03-29T11:40:19.083 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.071+0000 7f704a940640 1 librados: starting objecter 2024-03-29T11:40:19.083 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.071+0000 7f704a940640 1 Processor -- start 2024-03-29T11:40:19.083 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.071+0000 7f704a940640 1 -- start start 2024-03-29T11:40:19.083 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.071+0000 7f704a940640 1 librados: setting wanted keys 2024-03-29T11:40:19.084 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.071+0000 7f704a940640 1 librados: calling monclient init 2024-03-29T11:40:19.084 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.072+0000 7f704a940640 10 monclient: init 2024-03-29T11:40:19.084 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.072+0000 7f704a940640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:19.084 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.073+0000 7f704a940640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:40:19.084 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.073+0000 7f704a940640 10 monclient(hunting): picked mon.noname-c con 0x7f7044071c30 addr v1:172.21.15.72:6790/0 2024-03-29T11:40:19.084 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.073+0000 7f704a940640 10 monclient(hunting): picked mon.noname-a con 0x7f70440a9750 addr v1:172.21.15.72:6789/0 2024-03-29T11:40:19.084 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.073+0000 7f704a940640 10 monclient(hunting): picked mon.noname-b con 0x7f704413e1e0 addr v1:172.21.15.89:6789/0 2024-03-29T11:40:19.084 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.073+0000 7f704a940640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70441409d0 con 0x7f70440a9750 2024-03-29T11:40:19.084 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.073+0000 7f70437fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f70440a9750 0x7f70441410b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:36928/0 (socket says 172.21.15.72:36928) 2024-03-29T11:40:19.084 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.073+0000 7f70437fe640 1 -- 172.21.15.72:0/2474156090 learned_addr learned my addr 172.21.15.72:0/2474156090 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:40:19.084 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.073+0000 7f704a940640 1 -- 172.21.15.72:0/2474156090 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70441dbc90 con 0x7f704413e1e0 2024-03-29T11:40:19.084 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.073+0000 7f704a940640 1 -- 172.21.15.72:0/2474156090 --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70441427d0 con 0x7f7044071c30 2024-03-29T11:40:19.084 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.073+0000 7f704a940640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:19.084 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.073+0000 7f704a940640 10 monclient(hunting): authenticate will time out at 1178.361586s 2024-03-29T11:40:19.084 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.073+0000 7f7040ff9640 10 client.?.objecter ms_handle_connect 0x7f70440a9750 2024-03-29T11:40:19.084 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.073+0000 7f7040ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:19.084 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.073+0000 7f7040ff9640 10 client.?.objecter ms_handle_connect 0x7f7044071c30 2024-03-29T11:40:19.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.073+0000 7f7040ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:19.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.074+0000 7f7040ff9640 1 -- 172.21.15.72:0/2474156090 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4193771664 0 0) 0x7f70441427d0 con 0x7f7044071c30 2024-03-29T11:40:19.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.074+0000 7f7040ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:40:19.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.074+0000 7f7040ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:19.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.074+0000 7f7040ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:19.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.074+0000 7f7040ff9640 10 monclient(hunting): my global_id is 4430 2024-03-29T11:40:19.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.074+0000 7f7040ff9640 1 -- 172.21.15.72:0/2474156090 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7030004eb0 con 0x7f7044071c30 2024-03-29T11:40:19.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.074+0000 7f7040ff9640 1 -- 172.21.15.72:0/2474156090 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 482878411 0 0) 0x7f70441409d0 con 0x7f70440a9750 2024-03-29T11:40:19.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.074+0000 7f7040ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:40:19.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.074+0000 7f7040ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:19.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.074+0000 7f7040ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:19.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.074+0000 7f7040ff9640 10 monclient(hunting): my global_id is 4416 2024-03-29T11:40:19.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.074+0000 7f7040ff9640 1 -- 172.21.15.72:0/2474156090 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70441427d0 con 0x7f70440a9750 2024-03-29T11:40:19.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.074+0000 7f7040ff9640 10 client.?.objecter ms_handle_connect 0x7f704413e1e0 2024-03-29T11:40:19.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.074+0000 7f7040ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:19.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.074+0000 7f7040ff9640 1 -- 172.21.15.72:0/2474156090 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2044999164 0 0) 0x7f70441dbc90 con 0x7f704413e1e0 2024-03-29T11:40:19.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.074+0000 7f7040ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:40:19.086 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.074+0000 7f7040ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:19.086 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.074+0000 7f7040ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:19.086 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.075+0000 7f7040ff9640 10 monclient(hunting): my global_id is 4387 2024-03-29T11:40:19.086 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.075+0000 7f7040ff9640 1 -- 172.21.15.72:0/2474156090 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70441409d0 con 0x7f704413e1e0 2024-03-29T11:40:19.086 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.075+0000 7f7040ff9640 1 -- 172.21.15.72:0/2474156090 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3161272178 0 0) 0x7f70441427d0 con 0x7f70440a9750 2024-03-29T11:40:19.086 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.075+0000 7f7040ff9640 1 -- 172.21.15.72:0/2474156090 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70441dbc90 con 0x7f70440a9750 2024-03-29T11:40:19.086 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.075+0000 7f7040ff9640 1 -- 172.21.15.72:0/2474156090 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f7038001690 con 0x7f70440a9750 2024-03-29T11:40:19.086 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.075+0000 7f7040ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:19.086 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.075+0000 7f7040ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:40:19.086 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.075+0000 7f7040ff9640 10 monclient(hunting): dump: 2024-03-29T11:40:19.086 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:40:19.086 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:40:19.087 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:19.087 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:19.087 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:19.087 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:19.087 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:40:19.087 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:40:19.087 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:40:19.087 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:19.087 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.075+0000 7f7040ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:19.087 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.075+0000 7f7040ff9640 1 -- 172.21.15.72:0/2474156090 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1321606329 0 0) 0x7f7030004eb0 con 0x7f7044071c30 2024-03-29T11:40:19.087 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.075+0000 7f7040ff9640 1 -- 172.21.15.72:0/2474156090 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70441427d0 con 0x7f7044071c30 2024-03-29T11:40:19.087 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.075+0000 7f7040ff9640 1 -- 172.21.15.72:0/2474156090 <== mon.0 v1:172.21.15.72:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2062160876 0 0) 0x7f70441dbc90 con 0x7f70440a9750 2024-03-29T11:40:19.087 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.076+0000 7f7040ff9640 1 -- 172.21.15.72:0/2474156090 >> v1:172.21.15.72:6790/0 conn(0x7f7044071c30 legacy=0x7f70441db330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:19.087 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.076+0000 7f7040ff9640 1 -- 172.21.15.72:0/2474156090 >> v1:172.21.15.89:6789/0 conn(0x7f704413e1e0 legacy=0x7f70441417c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:19.087 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.076+0000 7f7040ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:40:19.087 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.076+0000 7f7040ff9640 1 monclient: found mon.a 2024-03-29T11:40:19.088 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.076+0000 7f7040ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:19.088 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.076+0000 7f7040ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:40:19.088 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.076+0000 7f7040ff9640 1 -- 172.21.15.72:0/2474156090 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7044142960 con 0x7f70440a9750 2024-03-29T11:40:19.088 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.076+0000 7f7040ff9640 10 monclient: _finish_auth 0 2024-03-29T11:40:19.088 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.076+0000 7f7040ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:40:19.088 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.076+0000 7f7040ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:19.088 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.076+0000 7f704a940640 5 monclient: authenticate success, global_id 4416 2024-03-29T11:40:19.088 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.076+0000 7f704a940640 10 monclient: _renew_subs 2024-03-29T11:40:19.088 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.076+0000 7f704a940640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:40:19.088 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.076+0000 7f704a940640 1 -- 172.21.15.72:0/2474156090 --> v1:172.21.15.72:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f704413f970 con 0x7f70440a9750 2024-03-29T11:40:19.088 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.076+0000 7f704a940640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:19.088 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.076+0000 7f704a940640 10 monclient: _renew_subs 2024-03-29T11:40:19.088 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.076+0000 7f704a940640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:40:19.088 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.076+0000 7f704a940640 1 -- 172.21.15.72:0/2474156090 --> v1:172.21.15.72:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f704413fc20 con 0x7f70440a9750 2024-03-29T11:40:19.088 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.076+0000 7f704a940640 1 librados: init done 2024-03-29T11:40:19.088 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.077+0000 7f7040ff9640 1 -- 172.21.15.72:0/2474156090 <== mon.0 v1:172.21.15.72:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70380038a0 con 0x7f70440a9750 2024-03-29T11:40:19.088 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.077+0000 7f7040ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:19.089 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.077+0000 7f7040ff9640 1 -- 172.21.15.72:0/2474156090 <== mon.0 v1:172.21.15.72:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f7038003ad0 con 0x7f70440a9750 2024-03-29T11:40:19.089 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.077+0000 7f7040ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:19.089 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.077+0000 7f7040ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:40:19.089 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.077+0000 7f7040ff9640 10 monclient: dump: 2024-03-29T11:40:19.089 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:40:19.089 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:40:19.089 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:19.089 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:19.089 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:19.089 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:19.089 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:40:19.089 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:40:19.089 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:40:19.089 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:19.089 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.077+0000 7f70427fc640 4 set_mon_vals no callback set 2024-03-29T11:40:19.089 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.077+0000 7f704a940640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:19.090 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.078+0000 7f704a940640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:19.090 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.078+0000 7f704a940640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:40:19.090 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.078+0000 7f704a940640 1 -- 172.21.15.72:0/2474156090 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7024005080 con 0x7f70440a9750 2024-03-29T11:40:19.090 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.079+0000 7f7040ff9640 1 -- 172.21.15.72:0/2474156090 <== mon.0 v1:172.21.15.72:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f7038003c30 con 0x7f70440a9750 2024-03-29T11:40:19.090 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.079+0000 7f7040ff9640 1 -- 172.21.15.72:0/2474156090 <== mon.0 v1:172.21.15.72:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f70380848c0 con 0x7f70440a9750 2024-03-29T11:40:19.090 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.079+0000 7f7040ff9640 10 client.4416.objecter ms_dispatch 0x7f7044000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:19.090 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.079+0000 7f7040ff9640 3 client.4416.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:19.090 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.079+0000 7f7040ff9640 3 client.4416.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:19.090 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.079+0000 7f7040ff9640 20 client.4416.objecter dump_active .. 0 homeless 2024-03-29T11:40:19.090 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.079+0000 7f7043fff640 10 monclient: get_auth_request con 0x7f703006dd50 auth_method 0 2024-03-29T11:40:19.090 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.079+0000 7f7040ff9640 10 client.4416.objecter ms_handle_connect 0x7f703006dd50 2024-03-29T11:40:19.091 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.087+0000 7f7040ff9640 1 -- 172.21.15.72:0/2474156090 <== mon.0 v1:172.21.15.72:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f70380501a0 con 0x7f70440a9750 2024-03-29T11:40:19.091 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.087+0000 7f7040ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:19.091 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.087+0000 7f7040ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:19.454 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.452+0000 7f704a940640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:19.454 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.452+0000 7f704a940640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:19.454 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.452+0000 7f704a940640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:40:19.454 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.452+0000 7f704a940640 1 -- 172.21.15.72:0/2474156090 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7024005330 con 0x7f70440a9750 2024-03-29T11:40:19.454 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.453+0000 7f7040ff9640 1 -- 172.21.15.72:0/2474156090 <== mon.0 v1:172.21.15.72:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f70380520d0 con 0x7f70440a9750 2024-03-29T11:40:19.454 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.453+0000 7f7040ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:19.455 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.453+0000 7f7040ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:19.455 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:40:19.455 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:19.458 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.456+0000 7f704a940640 10 librados: watch_flush enter 2024-03-29T11:40:19.458 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.456+0000 7f704a940640 10 librados: watch_flush exit 2024-03-29T11:40:19.458 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.456+0000 7f704a940640 20 client.4416.objecter shutdown clearing up homeless session... 2024-03-29T11:40:19.458 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.456+0000 7f704a940640 10 client.4416.objecter successfully canceled tick 2024-03-29T11:40:19.458 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.457+0000 7f704a940640 1 -- 172.21.15.72:0/2474156090 >> v1:172.21.15.72:6816/1578866102 conn(0x7f703006dd50 legacy=0x7f7030070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:19.458 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.457+0000 7f704a940640 10 monclient: shutdown 2024-03-29T11:40:19.458 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.457+0000 7f704a940640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:19.459 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.457+0000 7f704a940640 1 -- 172.21.15.72:0/2474156090 >> v1:172.21.15.72:6789/0 conn(0x7f70440a9750 legacy=0x7f70441410b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:19.459 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.457+0000 7f704a940640 1 -- 172.21.15.72:0/2474156090 shutdown_connections 2024-03-29T11:40:19.459 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.457+0000 7f704a940640 1 -- 172.21.15.72:0/2474156090 >> 172.21.15.72:0/2474156090 conn(0x7f704406b660 msgr2=0x7f704406ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:19.459 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.457+0000 7f704a940640 1 -- 172.21.15.72:0/2474156090 shutdown_connections 2024-03-29T11:40:19.459 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.457+0000 7f704a940640 1 -- 172.21.15.72:0/2474156090 wait complete. 2024-03-29T11:40:19.459 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:19.457+0000 7f704a940640 1 librados: shutdown 2024-03-29T11:40:19.473 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:26.280 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:26.435 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.433+0000 7fd5c359e640 10 monclient: get_monmap_and_config 2024-03-29T11:40:26.435 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.433+0000 7fd5c359e640 10 monclient: build_initial_monmap 2024-03-29T11:40:26.436 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.433+0000 7fd5c359e640 10 monclient: monmap: 2024-03-29T11:40:26.436 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:40:26.436 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:40:26.436 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:40:26.434745+0000 2024-03-29T11:40:26.436 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:40:26.434745+0000 2024-03-29T11:40:26.436 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:26.436 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:26.436 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:40:26.436 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:40:26.436 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:40:26.436 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:26.437 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.435+0000 7fd5c359e640 1 Processor -- start 2024-03-29T11:40:26.437 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.435+0000 7fd5c359e640 1 -- start start 2024-03-29T11:40:26.437 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.435+0000 7fd5c359e640 10 monclient: init 2024-03-29T11:40:26.440 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.436+0000 7fd5c359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:26.440 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.436+0000 7fd5c359e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:40:26.440 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.436+0000 7fd5c359e640 10 monclient(hunting): picked mon.noname-a con 0x7fd5c40b2800 addr v1:172.21.15.72:6789/0 2024-03-29T11:40:26.440 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.436+0000 7fd5c359e640 10 monclient(hunting): picked mon.noname-b con 0x7fd5c4142b20 addr v1:172.21.15.72:6790/0 2024-03-29T11:40:26.440 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.436+0000 7fd5c359e640 10 monclient(hunting): picked mon.noname-c con 0x7fd5c413c040 addr v1:172.21.15.89:6789/0 2024-03-29T11:40:26.441 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.436+0000 7fd5c359e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5c40aad10 con 0x7fd5c40b2800 2024-03-29T11:40:26.441 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.436+0000 7fd5c359e640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5c4141f30 con 0x7fd5c413c040 2024-03-29T11:40:26.441 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.436+0000 7fd5c359e640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5c40aaf60 con 0x7fd5c4142b20 2024-03-29T11:40:26.441 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.436+0000 7fd5c359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:26.441 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.436+0000 7fd5c359e640 10 monclient(hunting): authenticate will time out at 1185.724460s 2024-03-29T11:40:26.443 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.440+0000 7fd5c2d9d640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fd5c413c040 0x7fd5c414d7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:56185/0 (socket says 172.21.15.72:56185) 2024-03-29T11:40:26.443 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.442+0000 7fd5c2d9d640 1 -- 172.21.15.72:0/3488811295 learned_addr learned my addr 172.21.15.72:0/3488811295 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:40:26.447 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.442+0000 7fd5c159a640 1 -- 172.21.15.72:0/3488811295 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1568429917 0 0) 0x7fd5c40aad10 con 0x7fd5c40b2800 2024-03-29T11:40:26.447 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.442+0000 7fd5c159a640 10 monclient(hunting): _negotiate 2024-03-29T11:40:26.447 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.442+0000 7fd5c159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:26.447 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.442+0000 7fd5c159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:26.447 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.442+0000 7fd5c159a640 10 monclient(hunting): my global_id is 4419 2024-03-29T11:40:26.447 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.442+0000 7fd5c159a640 1 -- 172.21.15.72:0/3488811295 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5ac003520 con 0x7fd5c40b2800 2024-03-29T11:40:26.447 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.442+0000 7fd5c159a640 1 -- 172.21.15.72:0/3488811295 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 524047703 0 0) 0x7fd5c4141f30 con 0x7fd5c413c040 2024-03-29T11:40:26.447 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.442+0000 7fd5c159a640 10 monclient(hunting): _negotiate 2024-03-29T11:40:26.447 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.442+0000 7fd5c159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:26.448 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.442+0000 7fd5c159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:26.448 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.442+0000 7fd5c159a640 10 monclient(hunting): my global_id is 4390 2024-03-29T11:40:26.448 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.442+0000 7fd5c159a640 1 -- 172.21.15.72:0/3488811295 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5c40aad10 con 0x7fd5c413c040 2024-03-29T11:40:26.448 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.443+0000 7fd5c159a640 1 -- 172.21.15.72:0/3488811295 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3505639708 0 0) 0x7fd5ac003520 con 0x7fd5c40b2800 2024-03-29T11:40:26.448 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.443+0000 7fd5c159a640 1 -- 172.21.15.72:0/3488811295 >> v1:172.21.15.72:6790/0 conn(0x7fd5c4142b20 legacy=0x7fd5c413b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:40:26.448 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.443+0000 7fd5c159a640 1 -- 172.21.15.72:0/3488811295 >> v1:172.21.15.89:6789/0 conn(0x7fd5c413c040 legacy=0x7fd5c414d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:26.448 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.443+0000 7fd5c159a640 10 monclient: _finish_hunting 0 2024-03-29T11:40:26.448 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.443+0000 7fd5c159a640 1 monclient: found mon.noname-a 2024-03-29T11:40:26.448 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.443+0000 7fd5c159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:26.448 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.443+0000 7fd5c159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.72:6789/0 2024-03-29T11:40:26.448 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.443+0000 7fd5c159a640 1 -- 172.21.15.72:0/3488811295 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5c40b3900 con 0x7fd5c40b2800 2024-03-29T11:40:26.449 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.443+0000 7fd5c159a640 10 monclient: _finish_auth 0 2024-03-29T11:40:26.449 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.443+0000 7fd5c159a640 10 monclient: _check_auth_tickets 2024-03-29T11:40:26.449 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.443+0000 7fd5c159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:26.449 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.443+0000 7fd5c159a640 1 -- 172.21.15.72:0/3488811295 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fd5b4001690 con 0x7fd5c40b2800 2024-03-29T11:40:26.449 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.443+0000 7fd5c159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:26.449 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.443+0000 7fd5c159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:40:26.449 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.443+0000 7fd5c159a640 10 monclient: dump: 2024-03-29T11:40:26.449 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:40:26.449 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:40:26.449 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:26.449 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:26.449 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:26.449 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:26.449 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:40:26.449 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:40:26.450 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:40:26.450 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:26.450 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.443+0000 7fd5c359e640 5 monclient: authenticate success, global_id 4419 2024-03-29T11:40:26.450 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.443+0000 7fd5c359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:26.450 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.443+0000 7fd5c159a640 1 -- 172.21.15.72:0/3488811295 <== mon.0 v1:172.21.15.72:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5b40017f0 con 0x7fd5c40b2800 2024-03-29T11:40:26.450 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.443+0000 7fd5c159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:26.450 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.443+0000 7fd5c359e640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:26.450 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.443+0000 7fd5c359e640 4 set_mon_vals no callback set 2024-03-29T11:40:26.450 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.444+0000 7fd5c359e640 10 monclient: shutdown 2024-03-29T11:40:26.450 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.444+0000 7fd5c359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:26.450 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.444+0000 7fd5c359e640 1 -- 172.21.15.72:0/3488811295 >> v1:172.21.15.72:6789/0 conn(0x7fd5c40b2800 legacy=0x7fd5c4142410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:26.452 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.446+0000 7fd5c359e640 1 -- 172.21.15.72:0/3488811295 shutdown_connections 2024-03-29T11:40:26.452 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.446+0000 7fd5c359e640 1 -- 172.21.15.72:0/3488811295 >> 172.21.15.72:0/3488811295 conn(0x7fd5c406d0e0 msgr2=0x7fd5c4066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:26.452 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.447+0000 7fd5c359e640 1 -- 172.21.15.72:0/3488811295 shutdown_connections 2024-03-29T11:40:26.452 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.447+0000 7fd5c359e640 1 -- 172.21.15.72:0/3488811295 wait complete. 2024-03-29T11:40:26.452 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.448+0000 7fd5c359e640 10 monclient: build_initial_monmap 2024-03-29T11:40:26.452 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.448+0000 7fd5c359e640 10 monclient: monmap: 2024-03-29T11:40:26.452 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:40:26.452 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:26.453 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:40:26.453 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:40:26.453 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:26.453 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:26.453 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:40:26.453 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:40:26.453 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:40:26.453 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:26.453 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.449+0000 7fd5c359e640 1 librados: starting msgr at 2024-03-29T11:40:26.453 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.449+0000 7fd5c359e640 1 librados: starting objecter 2024-03-29T11:40:26.453 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.449+0000 7fd5c359e640 1 Processor -- start 2024-03-29T11:40:26.453 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.449+0000 7fd5c359e640 1 -- start start 2024-03-29T11:40:26.453 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.449+0000 7fd5c359e640 1 librados: setting wanted keys 2024-03-29T11:40:26.453 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.449+0000 7fd5c359e640 1 librados: calling monclient init 2024-03-29T11:40:26.453 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.449+0000 7fd5c359e640 10 monclient: init 2024-03-29T11:40:26.454 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.450+0000 7fd5c359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:26.454 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.450+0000 7fd5c359e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:40:26.454 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.450+0000 7fd5c359e640 10 monclient(hunting): picked mon.noname-a con 0x7fd5c40b2800 addr v1:172.21.15.72:6789/0 2024-03-29T11:40:26.454 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.450+0000 7fd5c359e640 10 monclient(hunting): picked mon.noname-c con 0x7fd5c413c040 addr v1:172.21.15.72:6790/0 2024-03-29T11:40:26.456 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.453+0000 7fd5c359e640 10 monclient(hunting): picked mon.noname-b con 0x7fd5c4142b20 addr v1:172.21.15.89:6789/0 2024-03-29T11:40:26.456 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.454+0000 7fd5c2d9d640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fd5c4142b20 0x7fd5c41e6d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:56247/0 (socket says 172.21.15.72:56247) 2024-03-29T11:40:26.456 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.454+0000 7fd5c2d9d640 1 -- 172.21.15.72:0/3779897406 learned_addr learned my addr 172.21.15.72:0/3779897406 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:40:26.456 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.454+0000 7fd5c359e640 1 -- 172.21.15.72:0/3779897406 --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5c41db530 con 0x7fd5c40b2800 2024-03-29T11:40:26.456 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.454+0000 7fd5c359e640 1 -- 172.21.15.72:0/3779897406 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5c41dc3b0 con 0x7fd5c4142b20 2024-03-29T11:40:26.456 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.454+0000 7fd5c359e640 1 -- 172.21.15.72:0/3779897406 --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5c4146130 con 0x7fd5c413c040 2024-03-29T11:40:26.456 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.454+0000 7fd5c359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:26.457 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.454+0000 7fd5b2ffd640 10 client.?.objecter ms_handle_connect 0x7fd5c4142b20 2024-03-29T11:40:26.457 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.454+0000 7fd5b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:26.457 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.455+0000 7fd5c359e640 10 monclient(hunting): authenticate will time out at 1185.743062s 2024-03-29T11:40:26.457 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.455+0000 7fd5b2ffd640 1 -- 172.21.15.72:0/3779897406 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3521435928 0 0) 0x7fd5c41dc3b0 con 0x7fd5c4142b20 2024-03-29T11:40:26.457 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.455+0000 7fd5b2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:26.458 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.455+0000 7fd5b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:26.458 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.455+0000 7fd5b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:26.458 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.455+0000 7fd5b2ffd640 10 monclient(hunting): my global_id is 4393 2024-03-29T11:40:26.458 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.455+0000 7fd5b2ffd640 1 -- 172.21.15.72:0/3779897406 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5a4004eb0 con 0x7fd5c4142b20 2024-03-29T11:40:26.458 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.456+0000 7fd5b2ffd640 10 client.?.objecter ms_handle_connect 0x7fd5c40b2800 2024-03-29T11:40:26.458 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.456+0000 7fd5b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:26.458 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.456+0000 7fd5b2ffd640 1 -- 172.21.15.72:0/3779897406 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3586476803 0 0) 0x7fd5c41db530 con 0x7fd5c40b2800 2024-03-29T11:40:26.458 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.456+0000 7fd5b2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:26.458 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.456+0000 7fd5b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:26.458 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.456+0000 7fd5b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:26.458 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.456+0000 7fd5b2ffd640 10 monclient(hunting): my global_id is 4422 2024-03-29T11:40:26.458 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.456+0000 7fd5b2ffd640 1 -- 172.21.15.72:0/3779897406 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5c41dc3b0 con 0x7fd5c40b2800 2024-03-29T11:40:26.458 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.456+0000 7fd5b2ffd640 10 client.?.objecter ms_handle_connect 0x7fd5c413c040 2024-03-29T11:40:26.459 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.456+0000 7fd5b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:26.463 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.456+0000 7fd5b2ffd640 1 -- 172.21.15.72:0/3779897406 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1204673765 0 0) 0x7fd5a4004eb0 con 0x7fd5c4142b20 2024-03-29T11:40:26.463 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.456+0000 7fd5b2ffd640 1 -- 172.21.15.72:0/3779897406 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5c41db530 con 0x7fd5c4142b20 2024-03-29T11:40:26.463 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.457+0000 7fd5b2ffd640 1 -- 172.21.15.72:0/3779897406 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 615010788 0 0) 0x7fd5c4146130 con 0x7fd5c413c040 2024-03-29T11:40:26.463 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.457+0000 7fd5b2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:26.463 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.457+0000 7fd5b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:26.463 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.457+0000 7fd5b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:26.463 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.457+0000 7fd5b2ffd640 10 monclient(hunting): my global_id is 4433 2024-03-29T11:40:26.463 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.457+0000 7fd5b2ffd640 1 -- 172.21.15.72:0/3779897406 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5a4004eb0 con 0x7fd5c413c040 2024-03-29T11:40:26.463 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.457+0000 7fd5b2ffd640 1 -- 172.21.15.72:0/3779897406 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fd5b8001950 con 0x7fd5c4142b20 2024-03-29T11:40:26.463 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.457+0000 7fd5b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:26.464 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.457+0000 7fd5b2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:26.464 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.457+0000 7fd5b2ffd640 10 monclient(hunting): dump: 2024-03-29T11:40:26.464 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:40:26.464 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:40:26.464 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:26.464 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:26.464 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:26.464 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:26.464 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:40:26.464 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:40:26.464 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:40:26.464 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:26.464 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.457+0000 7fd5b2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:26.464 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.458+0000 7fd5b2ffd640 1 -- 172.21.15.72:0/3779897406 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3272114469 0 0) 0x7fd5c41dc3b0 con 0x7fd5c40b2800 2024-03-29T11:40:26.464 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.458+0000 7fd5b2ffd640 1 -- 172.21.15.72:0/3779897406 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5c4146130 con 0x7fd5c40b2800 2024-03-29T11:40:26.465 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.458+0000 7fd5b2ffd640 1 -- 172.21.15.72:0/3779897406 <== mon.1 v1:172.21.15.89:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1921342901 0 0) 0x7fd5c41db530 con 0x7fd5c4142b20 2024-03-29T11:40:26.465 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.458+0000 7fd5b2ffd640 1 -- 172.21.15.72:0/3779897406 >> v1:172.21.15.72:6790/0 conn(0x7fd5c413c040 legacy=0x7fd5c4070970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:26.465 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.458+0000 7fd5b2ffd640 1 -- 172.21.15.72:0/3779897406 >> v1:172.21.15.72:6789/0 conn(0x7fd5c40b2800 legacy=0x7fd5c41dbca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:26.465 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.458+0000 7fd5b2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:40:26.465 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.458+0000 7fd5b2ffd640 1 monclient: found mon.b 2024-03-29T11:40:26.465 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.458+0000 7fd5b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:26.465 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.458+0000 7fd5b2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:40:26.465 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.458+0000 7fd5b2ffd640 1 -- 172.21.15.72:0/3779897406 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5c406de90 con 0x7fd5c4142b20 2024-03-29T11:40:26.465 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.458+0000 7fd5b2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:40:26.465 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.458+0000 7fd5b2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:40:26.465 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.458+0000 7fd5b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:26.465 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.459+0000 7fd5c359e640 5 monclient: authenticate success, global_id 4393 2024-03-29T11:40:26.465 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.459+0000 7fd5c359e640 10 monclient: _renew_subs 2024-03-29T11:40:26.465 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.459+0000 7fd5c359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:40:26.465 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.459+0000 7fd5c359e640 1 -- 172.21.15.72:0/3779897406 --> v1:172.21.15.89:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5c41db780 con 0x7fd5c4142b20 2024-03-29T11:40:26.465 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.459+0000 7fd5c359e640 10 client.4393.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:26.466 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.459+0000 7fd5c359e640 10 monclient: _renew_subs 2024-03-29T11:40:26.466 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.459+0000 7fd5c359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:40:26.466 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.459+0000 7fd5c359e640 1 -- 172.21.15.72:0/3779897406 --> v1:172.21.15.89:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5c40717d0 con 0x7fd5c4142b20 2024-03-29T11:40:26.466 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.459+0000 7fd5c359e640 1 librados: init done 2024-03-29T11:40:26.466 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.459+0000 7fd5b2ffd640 1 -- 172.21.15.72:0/3779897406 <== mon.1 v1:172.21.15.89:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5b8001ab0 con 0x7fd5c4142b20 2024-03-29T11:40:26.466 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.459+0000 7fd5b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:26.466 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.459+0000 7fd5c0d99640 4 set_mon_vals no callback set 2024-03-29T11:40:26.466 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.461+0000 7fd5b2ffd640 1 -- 172.21.15.72:0/3779897406 <== mon.1 v1:172.21.15.89:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fd5b8004a90 con 0x7fd5c4142b20 2024-03-29T11:40:26.466 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.461+0000 7fd5b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:26.466 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.461+0000 7fd5b2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:40:26.466 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.461+0000 7fd5b2ffd640 10 monclient: dump: 2024-03-29T11:40:26.466 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:40:26.466 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:40:26.466 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:26.466 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:26.467 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:26.467 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:26.467 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:40:26.467 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:40:26.467 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:40:26.467 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:26.468 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.462+0000 7fd5c359e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:26.468 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.462+0000 7fd5c359e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:26.468 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.462+0000 7fd5c359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:40:26.468 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.462+0000 7fd5c359e640 1 -- 172.21.15.72:0/3779897406 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5a8005080 con 0x7fd5c4142b20 2024-03-29T11:40:26.469 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.463+0000 7fd5b2ffd640 1 -- 172.21.15.72:0/3779897406 <== mon.1 v1:172.21.15.89:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7fd5b801aaa0 con 0x7fd5c4142b20 2024-03-29T11:40:26.469 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.464+0000 7fd5b2ffd640 1 -- 172.21.15.72:0/3779897406 <== mon.1 v1:172.21.15.89:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7fd5b80849a0 con 0x7fd5c4142b20 2024-03-29T11:40:26.469 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.464+0000 7fd5b2ffd640 10 client.4393.objecter ms_dispatch 0x7fd5c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:26.469 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.464+0000 7fd5c259c640 10 monclient: get_auth_request con 0x7fd5a406dff0 auth_method 0 2024-03-29T11:40:26.469 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.464+0000 7fd5b2ffd640 3 client.4393.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:26.469 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.464+0000 7fd5b2ffd640 3 client.4393.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:26.469 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.464+0000 7fd5b2ffd640 20 client.4393.objecter dump_active .. 0 homeless 2024-03-29T11:40:26.469 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.464+0000 7fd5b2ffd640 10 client.4393.objecter ms_handle_connect 0x7fd5a406dff0 2024-03-29T11:40:26.470 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.469+0000 7fd5b2ffd640 1 -- 172.21.15.72:0/3779897406 <== mon.1 v1:172.21.15.89:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd5b8050280 con 0x7fd5c4142b20 2024-03-29T11:40:26.471 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.469+0000 7fd5b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:26.471 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.469+0000 7fd5b2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:26.850 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.848+0000 7fd5c359e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:26.850 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.848+0000 7fd5c359e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:26.850 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.848+0000 7fd5c359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:40:26.850 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.848+0000 7fd5c359e640 1 -- 172.21.15.72:0/3779897406 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5a8005330 con 0x7fd5c4142b20 2024-03-29T11:40:26.850 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.848+0000 7fd5b2ffd640 1 -- 172.21.15.72:0/3779897406 <== mon.1 v1:172.21.15.89:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7fd5b80521b0 con 0x7fd5c4142b20 2024-03-29T11:40:26.850 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.848+0000 7fd5b2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:26.850 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.848+0000 7fd5b2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:26.851 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:40:26.851 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:26.855 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.853+0000 7fd5c359e640 10 librados: watch_flush enter 2024-03-29T11:40:26.855 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.853+0000 7fd5c359e640 10 librados: watch_flush exit 2024-03-29T11:40:26.855 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.853+0000 7fd5c359e640 20 client.4393.objecter shutdown clearing up homeless session... 2024-03-29T11:40:26.855 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.853+0000 7fd5c359e640 10 client.4393.objecter successfully canceled tick 2024-03-29T11:40:26.855 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.853+0000 7fd5c359e640 1 -- 172.21.15.72:0/3779897406 >> v1:172.21.15.72:6816/1578866102 conn(0x7fd5a406dff0 legacy=0x7fd5a40704b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:26.855 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.853+0000 7fd5c359e640 10 monclient: shutdown 2024-03-29T11:40:26.855 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.853+0000 7fd5c359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:26.855 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.853+0000 7fd5c359e640 1 -- 172.21.15.72:0/3779897406 >> v1:172.21.15.89:6789/0 conn(0x7fd5c4142b20 legacy=0x7fd5c41e6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:26.855 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.853+0000 7fd5c359e640 1 -- 172.21.15.72:0/3779897406 shutdown_connections 2024-03-29T11:40:26.855 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.853+0000 7fd5c359e640 1 -- 172.21.15.72:0/3779897406 >> 172.21.15.72:0/3779897406 conn(0x7fd5c406d0e0 msgr2=0x7fd5c40b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:26.856 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.853+0000 7fd5c359e640 1 -- 172.21.15.72:0/3779897406 shutdown_connections 2024-03-29T11:40:26.856 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.853+0000 7fd5c359e640 1 -- 172.21.15.72:0/3779897406 wait complete. 2024-03-29T11:40:26.856 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:26.853+0000 7fd5c359e640 1 librados: shutdown 2024-03-29T11:40:26.871 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:33.679 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:33.816 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.814+0000 7f0b338dd640 10 monclient: get_monmap_and_config 2024-03-29T11:40:33.816 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.814+0000 7f0b338dd640 10 monclient: build_initial_monmap 2024-03-29T11:40:33.816 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.814+0000 7f0b338dd640 10 monclient: monmap: 2024-03-29T11:40:33.816 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:40:33.816 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:40:33.816 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:40:33.815316+0000 2024-03-29T11:40:33.816 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:40:33.815316+0000 2024-03-29T11:40:33.816 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:33.817 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:33.817 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:40:33.817 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:40:33.817 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:40:33.817 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:33.817 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.815+0000 7f0b338dd640 1 Processor -- start 2024-03-29T11:40:33.820 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.815+0000 7f0b338dd640 1 -- start start 2024-03-29T11:40:33.820 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.815+0000 7f0b338dd640 10 monclient: init 2024-03-29T11:40:33.820 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.816+0000 7f0b338dd640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:33.820 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.816+0000 7f0b338dd640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:40:33.820 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.816+0000 7f0b338dd640 10 monclient(hunting): picked mon.noname-c con 0x7f0b2c1457f0 addr v1:172.21.15.89:6789/0 2024-03-29T11:40:33.820 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.816+0000 7f0b338dd640 10 monclient(hunting): picked mon.noname-b con 0x7f0b2c13f650 addr v1:172.21.15.72:6790/0 2024-03-29T11:40:33.820 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.816+0000 7f0b338dd640 10 monclient(hunting): picked mon.noname-a con 0x7f0b2c0a9d70 addr v1:172.21.15.72:6789/0 2024-03-29T11:40:33.821 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.816+0000 7f0b338dd640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b2c0aaef0 con 0x7f0b2c0a9d70 2024-03-29T11:40:33.821 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.816+0000 7f0b338dd640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b2c145bd0 con 0x7f0b2c1457f0 2024-03-29T11:40:33.821 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.816+0000 7f0b338dd640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b2c145eb0 con 0x7f0b2c13f650 2024-03-29T11:40:33.821 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.816+0000 7f0b338dd640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:33.821 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.816+0000 7f0b338dd640 10 monclient(hunting): authenticate will time out at 1193.104597s 2024-03-29T11:40:33.821 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.817+0000 7f0b30e78640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f0b2c13f650 0x7f0b2c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:51030/0 (socket says 172.21.15.72:51030) 2024-03-29T11:40:33.821 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.817+0000 7f0b30e78640 1 -- 172.21.15.72:0/3800642116 learned_addr learned my addr 172.21.15.72:0/3800642116 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:40:33.821 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.817+0000 7f0b2bfff640 1 -- 172.21.15.72:0/3800642116 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1712367857 0 0) 0x7f0b2c145eb0 con 0x7f0b2c13f650 2024-03-29T11:40:33.821 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.817+0000 7f0b2bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:40:33.821 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.817+0000 7f0b2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:33.821 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.817+0000 7f0b2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:33.821 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.817+0000 7f0b2bfff640 10 monclient(hunting): my global_id is 4436 2024-03-29T11:40:33.821 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.817+0000 7f0b2bfff640 1 -- 172.21.15.72:0/3800642116 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b14003520 con 0x7f0b2c13f650 2024-03-29T11:40:33.821 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.817+0000 7f0b2bfff640 1 -- 172.21.15.72:0/3800642116 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3828426354 0 0) 0x7f0b2c0aaef0 con 0x7f0b2c0a9d70 2024-03-29T11:40:33.821 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.817+0000 7f0b2bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:40:33.822 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.817+0000 7f0b2bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:33.822 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.817+0000 7f0b2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:33.822 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.817+0000 7f0b2bfff640 10 monclient(hunting): my global_id is 4425 2024-03-29T11:40:33.822 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.817+0000 7f0b2bfff640 1 -- 172.21.15.72:0/3800642116 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b2c145eb0 con 0x7f0b2c0a9d70 2024-03-29T11:40:33.822 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.818+0000 7f0b2bfff640 1 -- 172.21.15.72:0/3800642116 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 304162300 0 0) 0x7f0b14003520 con 0x7f0b2c13f650 2024-03-29T11:40:33.822 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.818+0000 7f0b2bfff640 1 -- 172.21.15.72:0/3800642116 >> v1:172.21.15.89:6789/0 conn(0x7f0b2c1457f0 legacy=0x7f0b2c13ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:33.823 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.819+0000 7f0b2bfff640 1 -- 172.21.15.72:0/3800642116 >> v1:172.21.15.72:6789/0 conn(0x7f0b2c0a9d70 legacy=0x7f0b2c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:33.823 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.819+0000 7f0b2bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:40:33.823 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.819+0000 7f0b2bfff640 1 monclient: found mon.noname-b 2024-03-29T11:40:33.823 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.819+0000 7f0b2bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:33.824 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.819+0000 7f0b2bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.72:6790/0 2024-03-29T11:40:33.824 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.819+0000 7f0b2bfff640 1 -- 172.21.15.72:0/3800642116 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b2c146340 con 0x7f0b2c13f650 2024-03-29T11:40:33.824 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.819+0000 7f0b2bfff640 10 monclient: _finish_auth 0 2024-03-29T11:40:33.824 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.819+0000 7f0b2bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:40:33.824 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.819+0000 7f0b2bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:33.824 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.819+0000 7f0b2bfff640 1 -- 172.21.15.72:0/3800642116 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f0b24001690 con 0x7f0b2c13f650 2024-03-29T11:40:33.824 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.819+0000 7f0b2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:33.824 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.819+0000 7f0b2bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:33.824 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.819+0000 7f0b2bfff640 10 monclient: dump: 2024-03-29T11:40:33.824 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:40:33.824 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:40:33.824 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:33.824 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:33.824 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:33.824 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:33.825 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:40:33.825 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:40:33.825 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:40:33.825 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:33.825 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.819+0000 7f0b338dd640 5 monclient: authenticate success, global_id 4436 2024-03-29T11:40:33.825 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.819+0000 7f0b338dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:33.825 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.819+0000 7f0b2bfff640 1 -- 172.21.15.72:0/3800642116 <== mon.2 v1:172.21.15.72:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b240017f0 con 0x7f0b2c13f650 2024-03-29T11:40:33.825 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.819+0000 7f0b2bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:33.825 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.819+0000 7f0b338dd640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:33.825 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.819+0000 7f0b338dd640 4 set_mon_vals no callback set 2024-03-29T11:40:33.825 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.819+0000 7f0b2bfff640 1 -- 172.21.15.72:0/3800642116 <== mon.2 v1:172.21.15.72:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f0b24001c40 con 0x7f0b2c13f650 2024-03-29T11:40:33.825 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.819+0000 7f0b2bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:33.825 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.819+0000 7f0b2bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:33.825 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.819+0000 7f0b2bfff640 10 monclient: dump: 2024-03-29T11:40:33.825 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:40:33.826 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:40:33.826 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:33.826 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:33.826 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:33.826 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:33.826 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:40:33.826 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:40:33.826 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:40:33.826 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:33.826 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.820+0000 7f0b338dd640 10 monclient: shutdown 2024-03-29T11:40:33.826 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.820+0000 7f0b338dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:33.826 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.820+0000 7f0b338dd640 1 -- 172.21.15.72:0/3800642116 >> v1:172.21.15.72:6790/0 conn(0x7f0b2c13f650 legacy=0x7f0b2c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:33.826 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.821+0000 7f0b338dd640 1 -- 172.21.15.72:0/3800642116 shutdown_connections 2024-03-29T11:40:33.826 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.821+0000 7f0b338dd640 1 -- 172.21.15.72:0/3800642116 >> 172.21.15.72:0/3800642116 conn(0x7f0b2c066a00 msgr2=0x7f0b2c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:33.826 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.821+0000 7f0b338dd640 1 -- 172.21.15.72:0/3800642116 shutdown_connections 2024-03-29T11:40:33.827 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.821+0000 7f0b338dd640 1 -- 172.21.15.72:0/3800642116 wait complete. 2024-03-29T11:40:33.833 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.822+0000 7f0b338dd640 10 monclient: build_initial_monmap 2024-03-29T11:40:33.833 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.822+0000 7f0b338dd640 10 monclient: monmap: 2024-03-29T11:40:33.833 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:40:33.834 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:33.834 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:40:33.834 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:40:33.834 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:33.834 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:33.834 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:40:33.834 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:40:33.834 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:40:33.834 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:33.834 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.822+0000 7f0b338dd640 1 librados: starting msgr at 2024-03-29T11:40:33.834 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.823+0000 7f0b338dd640 1 librados: starting objecter 2024-03-29T11:40:33.834 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.823+0000 7f0b338dd640 1 Processor -- start 2024-03-29T11:40:33.834 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.823+0000 7f0b338dd640 1 -- start start 2024-03-29T11:40:33.834 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.823+0000 7f0b338dd640 1 librados: setting wanted keys 2024-03-29T11:40:33.835 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.823+0000 7f0b338dd640 1 librados: calling monclient init 2024-03-29T11:40:33.835 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.823+0000 7f0b338dd640 10 monclient: init 2024-03-29T11:40:33.835 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.824+0000 7f0b338dd640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:33.835 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.824+0000 7f0b338dd640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:40:33.835 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.824+0000 7f0b338dd640 10 monclient(hunting): picked mon.noname-b con 0x7f0b2c0a9d70 addr v1:172.21.15.89:6789/0 2024-03-29T11:40:33.835 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.824+0000 7f0b338dd640 10 monclient(hunting): picked mon.noname-c con 0x7f0b2c13f650 addr v1:172.21.15.72:6790/0 2024-03-29T11:40:33.835 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.825+0000 7f0b338dd640 10 monclient(hunting): picked mon.noname-a con 0x7f0b2c1457f0 addr v1:172.21.15.72:6789/0 2024-03-29T11:40:33.835 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.825+0000 7f0b338dd640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b2c06efa0 con 0x7f0b2c1457f0 2024-03-29T11:40:33.835 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.825+0000 7f0b338dd640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b2c142c50 con 0x7f0b2c0a9d70 2024-03-29T11:40:33.835 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.825+0000 7f0b338dd640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b2c142660 con 0x7f0b2c13f650 2024-03-29T11:40:33.835 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.825+0000 7f0b338dd640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:33.835 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.825+0000 7f0b338dd640 10 monclient(hunting): authenticate will time out at 1193.113055s 2024-03-29T11:40:33.835 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.825+0000 7f0b30e78640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f0b2c13f650 0x7f0b2c1376f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:51824/0 (socket says 172.21.15.72:51824) 2024-03-29T11:40:33.835 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.825+0000 7f0b30e78640 1 -- 172.21.15.72:0/3357269154 learned_addr learned my addr 172.21.15.72:0/3357269154 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:40:33.836 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.825+0000 7f0b29ffb640 10 client.?.objecter ms_handle_connect 0x7f0b2c13f650 2024-03-29T11:40:33.836 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.825+0000 7f0b29ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:33.836 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.825+0000 7f0b29ffb640 1 -- 172.21.15.72:0/3357269154 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1827270343 0 0) 0x7f0b2c142660 con 0x7f0b2c13f650 2024-03-29T11:40:33.836 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.825+0000 7f0b29ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:40:33.836 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.825+0000 7f0b29ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:33.836 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.825+0000 7f0b29ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:33.836 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.825+0000 7f0b29ffb640 10 monclient(hunting): my global_id is 4439 2024-03-29T11:40:33.836 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.825+0000 7f0b29ffb640 1 -- 172.21.15.72:0/3357269154 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b18004eb0 con 0x7f0b2c13f650 2024-03-29T11:40:33.836 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.825+0000 7f0b29ffb640 10 client.?.objecter ms_handle_connect 0x7f0b2c1457f0 2024-03-29T11:40:33.836 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.825+0000 7f0b29ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:33.836 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.826+0000 7f0b29ffb640 1 -- 172.21.15.72:0/3357269154 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4239433069 0 0) 0x7f0b18004eb0 con 0x7f0b2c13f650 2024-03-29T11:40:33.836 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.826+0000 7f0b29ffb640 1 -- 172.21.15.72:0/3357269154 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b2c142660 con 0x7f0b2c13f650 2024-03-29T11:40:33.836 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.826+0000 7f0b29ffb640 1 -- 172.21.15.72:0/3357269154 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f0b24001bc0 con 0x7f0b2c13f650 2024-03-29T11:40:33.836 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.826+0000 7f0b29ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:33.836 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.826+0000 7f0b29ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:33.837 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.826+0000 7f0b29ffb640 10 monclient(hunting): dump: 2024-03-29T11:40:33.837 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:40:33.837 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:40:33.837 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:33.837 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:33.837 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:33.837 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:33.837 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:40:33.837 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:40:33.837 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:40:33.837 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:33.837 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.826+0000 7f0b29ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:33.837 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.826+0000 7f0b29ffb640 1 -- 172.21.15.72:0/3357269154 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1940293956 0 0) 0x7f0b2c06efa0 con 0x7f0b2c1457f0 2024-03-29T11:40:33.837 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.826+0000 7f0b29ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:40:33.837 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.826+0000 7f0b29ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:33.837 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.826+0000 7f0b29ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:33.838 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.826+0000 7f0b29ffb640 10 monclient(hunting): my global_id is 4428 2024-03-29T11:40:33.838 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.826+0000 7f0b29ffb640 1 -- 172.21.15.72:0/3357269154 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b18004eb0 con 0x7f0b2c1457f0 2024-03-29T11:40:33.838 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.826+0000 7f0b29ffb640 1 -- 172.21.15.72:0/3357269154 <== mon.2 v1:172.21.15.72:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3113310385 0 0) 0x7f0b2c142660 con 0x7f0b2c13f650 2024-03-29T11:40:33.838 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.826+0000 7f0b29ffb640 1 -- 172.21.15.72:0/3357269154 >> v1:172.21.15.89:6789/0 conn(0x7f0b2c0a9d70 legacy=0x7f0b2c141f50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:40:33.838 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.827+0000 7f0b29ffb640 1 -- 172.21.15.72:0/3357269154 >> v1:172.21.15.72:6789/0 conn(0x7f0b2c1457f0 legacy=0x7f0b2c1e6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:33.838 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.827+0000 7f0b29ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:40:33.838 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.827+0000 7f0b29ffb640 1 monclient: found mon.c 2024-03-29T11:40:33.838 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.827+0000 7f0b29ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:33.838 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.827+0000 7f0b29ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:40:33.838 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.827+0000 7f0b29ffb640 1 -- 172.21.15.72:0/3357269154 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b2c142900 con 0x7f0b2c13f650 2024-03-29T11:40:33.838 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.827+0000 7f0b29ffb640 10 monclient: _finish_auth 0 2024-03-29T11:40:33.838 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.827+0000 7f0b29ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:40:33.838 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.827+0000 7f0b29ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:33.838 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.827+0000 7f0b338dd640 5 monclient: authenticate success, global_id 4439 2024-03-29T11:40:33.839 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.827+0000 7f0b338dd640 10 monclient: _renew_subs 2024-03-29T11:40:33.839 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.827+0000 7f0b338dd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:40:33.839 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.827+0000 7f0b338dd640 1 -- 172.21.15.72:0/3357269154 --> v1:172.21.15.72:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b2c137e00 con 0x7f0b2c13f650 2024-03-29T11:40:33.839 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.827+0000 7f0b338dd640 10 client.4439.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:33.839 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.827+0000 7f0b338dd640 10 monclient: _renew_subs 2024-03-29T11:40:33.839 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.827+0000 7f0b338dd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:40:33.839 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.827+0000 7f0b338dd640 1 -- 172.21.15.72:0/3357269154 --> v1:172.21.15.72:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b2c13b470 con 0x7f0b2c13f650 2024-03-29T11:40:33.839 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.827+0000 7f0b338dd640 1 librados: init done 2024-03-29T11:40:33.839 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.828+0000 7f0b29ffb640 1 -- 172.21.15.72:0/3357269154 <== mon.2 v1:172.21.15.72:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b24004c60 con 0x7f0b2c13f650 2024-03-29T11:40:33.839 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.828+0000 7f0b29ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:33.839 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.829+0000 7f0b29ffb640 1 -- 172.21.15.72:0/3357269154 <== mon.2 v1:172.21.15.72:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f0b240050b0 con 0x7f0b2c13f650 2024-03-29T11:40:33.839 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.829+0000 7f0b29ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:33.839 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.829+0000 7f0b29ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:33.839 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.829+0000 7f0b29ffb640 10 monclient: dump: 2024-03-29T11:40:33.839 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:40:33.840 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:40:33.840 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:33.840 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:33.840 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:33.840 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:33.840 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:40:33.840 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:40:33.840 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:40:33.840 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:33.840 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.829+0000 7f0b2b7fe640 4 set_mon_vals no callback set 2024-03-29T11:40:33.840 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.830+0000 7f0b338dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:33.840 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.830+0000 7f0b29ffb640 1 -- 172.21.15.72:0/3357269154 <== mon.2 v1:172.21.15.72:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f0b2401b080 con 0x7f0b2c13f650 2024-03-29T11:40:33.840 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.830+0000 7f0b338dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:33.840 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.830+0000 7f0b338dd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:40:33.840 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.830+0000 7f0b338dd640 1 -- 172.21.15.72:0/3357269154 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b0c005080 con 0x7f0b2c13f650 2024-03-29T11:40:33.841 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.831+0000 7f0b31679640 10 monclient: get_auth_request con 0x7f0b1806daf0 auth_method 0 2024-03-29T11:40:33.841 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.831+0000 7f0b29ffb640 1 -- 172.21.15.72:0/3357269154 <== mon.2 v1:172.21.15.72:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f0b24085cf0 con 0x7f0b2c13f650 2024-03-29T11:40:33.841 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.831+0000 7f0b29ffb640 10 client.4439.objecter ms_dispatch 0x7f0b2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:33.841 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.831+0000 7f0b29ffb640 3 client.4439.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:33.841 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.831+0000 7f0b29ffb640 3 client.4439.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:33.841 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.831+0000 7f0b29ffb640 20 client.4439.objecter dump_active .. 0 homeless 2024-03-29T11:40:33.841 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.831+0000 7f0b29ffb640 10 client.4439.objecter ms_handle_connect 0x7f0b1806daf0 2024-03-29T11:40:33.841 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.837+0000 7f0b29ffb640 1 -- 172.21.15.72:0/3357269154 <== mon.2 v1:172.21.15.72:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0b240505a0 con 0x7f0b2c13f650 2024-03-29T11:40:33.841 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.837+0000 7f0b29ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:33.842 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:33.837+0000 7f0b29ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:34.160 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:34.158+0000 7f0b338dd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:34.160 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:34.158+0000 7f0b338dd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:34.160 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:34.158+0000 7f0b338dd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:40:34.160 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:34.158+0000 7f0b338dd640 1 -- 172.21.15.72:0/3357269154 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b0c005330 con 0x7f0b2c13f650 2024-03-29T11:40:34.161 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:34.159+0000 7f0b29ffb640 1 -- 172.21.15.72:0/3357269154 <== mon.2 v1:172.21.15.72:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f0b240524d0 con 0x7f0b2c13f650 2024-03-29T11:40:34.161 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:34.159+0000 7f0b29ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:34.161 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:34.159+0000 7f0b29ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:34.164 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:40:34.165 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:34.169 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:34.166+0000 7f0b338dd640 10 librados: watch_flush enter 2024-03-29T11:40:34.169 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:34.166+0000 7f0b338dd640 10 librados: watch_flush exit 2024-03-29T11:40:34.169 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:34.166+0000 7f0b338dd640 20 client.4439.objecter shutdown clearing up homeless session... 2024-03-29T11:40:34.169 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:34.166+0000 7f0b338dd640 10 client.4439.objecter successfully canceled tick 2024-03-29T11:40:34.169 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:34.166+0000 7f0b338dd640 1 -- 172.21.15.72:0/3357269154 >> v1:172.21.15.72:6816/1578866102 conn(0x7f0b1806daf0 legacy=0x7f0b1806ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:34.169 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:34.166+0000 7f0b338dd640 10 monclient: shutdown 2024-03-29T11:40:34.169 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:34.166+0000 7f0b338dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:34.169 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:34.167+0000 7f0b338dd640 1 -- 172.21.15.72:0/3357269154 >> v1:172.21.15.72:6790/0 conn(0x7f0b2c13f650 legacy=0x7f0b2c1376f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:34.169 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:34.167+0000 7f0b338dd640 1 -- 172.21.15.72:0/3357269154 shutdown_connections 2024-03-29T11:40:34.169 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:34.167+0000 7f0b338dd640 1 -- 172.21.15.72:0/3357269154 >> 172.21.15.72:0/3357269154 conn(0x7f0b2c066a00 msgr2=0x7f0b2c0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:34.169 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:34.167+0000 7f0b338dd640 1 -- 172.21.15.72:0/3357269154 shutdown_connections 2024-03-29T11:40:34.170 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:34.167+0000 7f0b338dd640 1 -- 172.21.15.72:0/3357269154 wait complete. 2024-03-29T11:40:34.170 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:34.167+0000 7f0b338dd640 1 librados: shutdown 2024-03-29T11:40:34.186 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:40.994 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:41.126 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.124+0000 7f9e057b5640 10 monclient: get_monmap_and_config 2024-03-29T11:40:41.126 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.124+0000 7f9e057b5640 10 monclient: build_initial_monmap 2024-03-29T11:40:41.127 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.124+0000 7f9e057b5640 10 monclient: monmap: 2024-03-29T11:40:41.127 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:40:41.127 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:40:41.127 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:40:41.125587+0000 2024-03-29T11:40:41.127 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:40:41.125587+0000 2024-03-29T11:40:41.127 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:41.127 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:41.127 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:40:41.127 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:40:41.127 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:40:41.127 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:41.129 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.125+0000 7f9e057b5640 1 Processor -- start 2024-03-29T11:40:41.129 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.125+0000 7f9e057b5640 1 -- start start 2024-03-29T11:40:41.129 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.125+0000 7f9e057b5640 10 monclient: init 2024-03-29T11:40:41.129 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.126+0000 7f9e057b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:41.129 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.126+0000 7f9e057b5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:40:41.129 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.126+0000 7f9e057b5640 10 monclient(hunting): picked mon.noname-a con 0x7f9e001457f0 addr v1:172.21.15.72:6789/0 2024-03-29T11:40:41.129 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.126+0000 7f9e057b5640 10 monclient(hunting): picked mon.noname-b con 0x7f9e0013f650 addr v1:172.21.15.72:6790/0 2024-03-29T11:40:41.129 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.126+0000 7f9e057b5640 10 monclient(hunting): picked mon.noname-c con 0x7f9e000a9d70 addr v1:172.21.15.89:6789/0 2024-03-29T11:40:41.129 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.126+0000 7f9e057b5640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e000aaef0 con 0x7f9e001457f0 2024-03-29T11:40:41.130 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.126+0000 7f9e057b5640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e00145bd0 con 0x7f9e000a9d70 2024-03-29T11:40:41.130 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.126+0000 7f9e057b5640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e00145eb0 con 0x7f9e0013f650 2024-03-29T11:40:41.130 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.126+0000 7f9e057b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:41.130 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.126+0000 7f9e057b5640 10 monclient(hunting): authenticate will time out at 1200.414299s 2024-03-29T11:40:41.130 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.126+0000 7f9dfe7fc640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f9e0013f650 0x7f9e000b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:43040/0 (socket says 172.21.15.72:43040) 2024-03-29T11:40:41.130 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.126+0000 7f9dfe7fc640 1 -- 172.21.15.72:0/1989361411 learned_addr learned my addr 172.21.15.72:0/1989361411 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:40:41.130 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.126+0000 7f9dfdffb640 1 -- 172.21.15.72:0/1989361411 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2998906161 0 0) 0x7f9e00145eb0 con 0x7f9e0013f650 2024-03-29T11:40:41.131 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.126+0000 7f9dfdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:40:41.131 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.126+0000 7f9dfdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:41.131 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.126+0000 7f9dfdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:41.131 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.126+0000 7f9dfdffb640 10 monclient(hunting): my global_id is 4442 2024-03-29T11:40:41.131 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.126+0000 7f9dfdffb640 1 -- 172.21.15.72:0/1989361411 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9de8003520 con 0x7f9e0013f650 2024-03-29T11:40:41.131 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.126+0000 7f9dfdffb640 1 -- 172.21.15.72:0/1989361411 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1304347329 0 0) 0x7f9de8003520 con 0x7f9e0013f650 2024-03-29T11:40:41.131 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.126+0000 7f9dfdffb640 1 -- 172.21.15.72:0/1989361411 >> v1:172.21.15.89:6789/0 conn(0x7f9e000a9d70 legacy=0x7f9e0014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:40:41.131 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.126+0000 7f9dfdffb640 1 -- 172.21.15.72:0/1989361411 >> v1:172.21.15.72:6789/0 conn(0x7f9e001457f0 legacy=0x7f9e0013ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:41.132 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.128+0000 7f9dfdffb640 10 monclient: _finish_hunting 0 2024-03-29T11:40:41.133 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.128+0000 7f9dfdffb640 1 monclient: found mon.noname-b 2024-03-29T11:40:41.133 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.128+0000 7f9dfdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:41.133 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.128+0000 7f9dfdffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.72:6790/0 2024-03-29T11:40:41.133 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.128+0000 7f9dfdffb640 1 -- 172.21.15.72:0/1989361411 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e00146340 con 0x7f9e0013f650 2024-03-29T11:40:41.133 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.128+0000 7f9dfdffb640 10 monclient: _finish_auth 0 2024-03-29T11:40:41.133 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.128+0000 7f9dfdffb640 10 monclient: _check_auth_tickets 2024-03-29T11:40:41.133 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.128+0000 7f9dfdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:41.133 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.128+0000 7f9dfdffb640 1 -- 172.21.15.72:0/1989361411 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f9df4001690 con 0x7f9e0013f650 2024-03-29T11:40:41.133 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.128+0000 7f9dfdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:41.133 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.128+0000 7f9dfdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:41.133 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.128+0000 7f9dfdffb640 10 monclient: dump: 2024-03-29T11:40:41.133 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:40:41.133 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:40:41.133 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:41.133 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:41.134 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:41.134 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:41.134 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:40:41.134 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:40:41.134 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:40:41.134 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:41.134 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.128+0000 7f9e057b5640 5 monclient: authenticate success, global_id 4442 2024-03-29T11:40:41.134 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.128+0000 7f9e057b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:41.134 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.129+0000 7f9dfdffb640 1 -- 172.21.15.72:0/1989361411 <== mon.2 v1:172.21.15.72:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9df4001860 con 0x7f9e0013f650 2024-03-29T11:40:41.134 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.129+0000 7f9dfdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:41.134 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.129+0000 7f9dfdffb640 1 -- 172.21.15.72:0/1989361411 <== mon.2 v1:172.21.15.72:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f9df4001cb0 con 0x7f9e0013f650 2024-03-29T11:40:41.134 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.129+0000 7f9dfdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:41.134 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.129+0000 7f9dfdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:41.134 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.129+0000 7f9dfdffb640 10 monclient: dump: 2024-03-29T11:40:41.134 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:40:41.135 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:40:41.135 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:41.135 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:41.135 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:41.135 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:41.135 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:40:41.135 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:40:41.135 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:40:41.135 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:41.135 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.129+0000 7f9e057b5640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:41.135 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.129+0000 7f9e057b5640 4 set_mon_vals no callback set 2024-03-29T11:40:41.135 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.130+0000 7f9e057b5640 10 monclient: shutdown 2024-03-29T11:40:41.135 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.130+0000 7f9e057b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:41.135 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.130+0000 7f9e057b5640 1 -- 172.21.15.72:0/1989361411 >> v1:172.21.15.72:6790/0 conn(0x7f9e0013f650 legacy=0x7f9e000b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:41.135 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.130+0000 7f9e057b5640 1 -- 172.21.15.72:0/1989361411 shutdown_connections 2024-03-29T11:40:41.135 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.130+0000 7f9e057b5640 1 -- 172.21.15.72:0/1989361411 >> 172.21.15.72:0/1989361411 conn(0x7f9e00066a00 msgr2=0x7f9e00066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:41.136 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.130+0000 7f9e057b5640 1 -- 172.21.15.72:0/1989361411 shutdown_connections 2024-03-29T11:40:41.136 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.130+0000 7f9e057b5640 1 -- 172.21.15.72:0/1989361411 wait complete. 2024-03-29T11:40:41.140 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.131+0000 7f9e057b5640 10 monclient: build_initial_monmap 2024-03-29T11:40:41.140 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.131+0000 7f9e057b5640 10 monclient: monmap: 2024-03-29T11:40:41.141 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:40:41.141 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:41.141 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:40:41.141 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:40:41.141 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:41.141 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:41.141 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:40:41.141 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:40:41.141 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:40:41.141 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:41.141 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.132+0000 7f9e057b5640 1 librados: starting msgr at 2024-03-29T11:40:41.141 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.132+0000 7f9e057b5640 1 librados: starting objecter 2024-03-29T11:40:41.141 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.132+0000 7f9e057b5640 1 Processor -- start 2024-03-29T11:40:41.141 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.132+0000 7f9e057b5640 1 -- start start 2024-03-29T11:40:41.141 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.132+0000 7f9e057b5640 1 librados: setting wanted keys 2024-03-29T11:40:41.142 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.132+0000 7f9e057b5640 1 librados: calling monclient init 2024-03-29T11:40:41.142 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.132+0000 7f9e057b5640 10 monclient: init 2024-03-29T11:40:41.142 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.133+0000 7f9e057b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:41.142 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.133+0000 7f9e057b5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:40:41.142 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.134+0000 7f9e057b5640 10 monclient(hunting): picked mon.noname-a con 0x7f9e000a9d70 addr v1:172.21.15.72:6789/0 2024-03-29T11:40:41.142 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.134+0000 7f9e057b5640 10 monclient(hunting): picked mon.noname-b con 0x7f9e0013f650 addr v1:172.21.15.89:6789/0 2024-03-29T11:40:41.142 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.134+0000 7f9e057b5640 10 monclient(hunting): picked mon.noname-c con 0x7f9e001457f0 addr v1:172.21.15.72:6790/0 2024-03-29T11:40:41.142 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.134+0000 7f9e057b5640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e00142060 con 0x7f9e000a9d70 2024-03-29T11:40:41.142 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.134+0000 7f9e057b5640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e00132bd0 con 0x7f9e0013f650 2024-03-29T11:40:41.142 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.134+0000 7f9e057b5640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e00132eb0 con 0x7f9e001457f0 2024-03-29T11:40:41.142 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.134+0000 7f9e057b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:41.142 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.134+0000 7f9e057b5640 10 monclient(hunting): authenticate will time out at 1200.422323s 2024-03-29T11:40:41.142 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.134+0000 7f9dff7fe640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f9e001457f0 0x7f9e00133890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:43320/0 (socket says 172.21.15.72:43320) 2024-03-29T11:40:41.142 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.134+0000 7f9dff7fe640 1 -- 172.21.15.72:0/735612822 learned_addr learned my addr 172.21.15.72:0/735612822 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:40:41.143 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.134+0000 7f9de77fe640 10 client.?.objecter ms_handle_connect 0x7f9e001457f0 2024-03-29T11:40:41.143 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.134+0000 7f9de77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:41.143 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.134+0000 7f9de77fe640 1 -- 172.21.15.72:0/735612822 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3480308088 0 0) 0x7f9e00132eb0 con 0x7f9e001457f0 2024-03-29T11:40:41.143 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.134+0000 7f9de77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:40:41.143 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.134+0000 7f9de77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:41.143 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.134+0000 7f9de77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:41.143 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.134+0000 7f9de77fe640 10 monclient(hunting): my global_id is 4445 2024-03-29T11:40:41.143 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.134+0000 7f9de77fe640 1 -- 172.21.15.72:0/735612822 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dec004eb0 con 0x7f9e001457f0 2024-03-29T11:40:41.143 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.135+0000 7f9de77fe640 1 -- 172.21.15.72:0/735612822 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1350877051 0 0) 0x7f9dec004eb0 con 0x7f9e001457f0 2024-03-29T11:40:41.143 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.135+0000 7f9de77fe640 1 -- 172.21.15.72:0/735612822 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e00132eb0 con 0x7f9e001457f0 2024-03-29T11:40:41.143 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.135+0000 7f9de77fe640 1 -- 172.21.15.72:0/735612822 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f9df8001c60 con 0x7f9e001457f0 2024-03-29T11:40:41.143 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.135+0000 7f9de77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:41.143 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.135+0000 7f9de77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:41.143 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.135+0000 7f9de77fe640 10 monclient(hunting): dump: 2024-03-29T11:40:41.143 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:40:41.143 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:40:41.144 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:41.144 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:41.144 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:41.144 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:41.144 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:40:41.144 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:40:41.144 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:40:41.144 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:41.144 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.135+0000 7f9de77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:41.144 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.135+0000 7f9de77fe640 1 -- 172.21.15.72:0/735612822 <== mon.2 v1:172.21.15.72:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1563946429 0 0) 0x7f9e00132eb0 con 0x7f9e001457f0 2024-03-29T11:40:41.144 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.135+0000 7f9de77fe640 1 -- 172.21.15.72:0/735612822 >> v1:172.21.15.89:6789/0 conn(0x7f9e0013f650 legacy=0x7f9e00140db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:41.144 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.135+0000 7f9de77fe640 1 -- 172.21.15.72:0/735612822 >> v1:172.21.15.72:6789/0 conn(0x7f9e000a9d70 legacy=0x7f9e00131e80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:40:41.144 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.136+0000 7f9de77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:40:41.144 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.136+0000 7f9de77fe640 1 monclient: found mon.c 2024-03-29T11:40:41.144 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.136+0000 7f9de77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:41.145 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.136+0000 7f9de77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:40:41.145 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.136+0000 7f9de77fe640 1 -- 172.21.15.72:0/735612822 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e00133190 con 0x7f9e001457f0 2024-03-29T11:40:41.145 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.136+0000 7f9de77fe640 10 monclient: _finish_auth 0 2024-03-29T11:40:41.145 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.136+0000 7f9de77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:40:41.145 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.136+0000 7f9de77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:41.145 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.136+0000 7f9e057b5640 5 monclient: authenticate success, global_id 4445 2024-03-29T11:40:41.145 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.136+0000 7f9e057b5640 10 monclient: _renew_subs 2024-03-29T11:40:41.145 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.136+0000 7f9e057b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:40:41.145 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.136+0000 7f9e057b5640 1 -- 172.21.15.72:0/735612822 --> v1:172.21.15.72:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e00142af0 con 0x7f9e001457f0 2024-03-29T11:40:41.145 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.136+0000 7f9e057b5640 10 client.4445.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:41.145 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.136+0000 7f9e057b5640 10 monclient: _renew_subs 2024-03-29T11:40:41.145 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.136+0000 7f9e057b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:40:41.145 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.136+0000 7f9e057b5640 1 -- 172.21.15.72:0/735612822 --> v1:172.21.15.72:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e0012cdd0 con 0x7f9e001457f0 2024-03-29T11:40:41.145 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.136+0000 7f9e057b5640 1 librados: init done 2024-03-29T11:40:41.145 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.137+0000 7f9de77fe640 1 -- 172.21.15.72:0/735612822 <== mon.2 v1:172.21.15.72:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9df8003970 con 0x7f9e001457f0 2024-03-29T11:40:41.145 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.137+0000 7f9de77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:41.146 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.137+0000 7f9de77fe640 1 -- 172.21.15.72:0/735612822 <== mon.2 v1:172.21.15.72:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f9df8003d80 con 0x7f9e001457f0 2024-03-29T11:40:41.146 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.137+0000 7f9de77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:41.146 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.137+0000 7f9de77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:41.146 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.137+0000 7f9de77fe640 10 monclient: dump: 2024-03-29T11:40:41.146 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:40:41.146 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:40:41.146 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:41.146 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:41.146 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:41.146 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:41.146 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:40:41.146 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:40:41.146 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:40:41.146 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:41.146 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.137+0000 7f9dfd7fa640 4 set_mon_vals no callback set 2024-03-29T11:40:41.147 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.138+0000 7f9e057b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:41.147 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.138+0000 7f9e057b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:41.147 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.138+0000 7f9e057b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:40:41.147 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.138+0000 7f9e057b5640 1 -- 172.21.15.72:0/735612822 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9de0005080 con 0x7f9e001457f0 2024-03-29T11:40:41.148 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.139+0000 7f9de77fe640 1 -- 172.21.15.72:0/735612822 <== mon.2 v1:172.21.15.72:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f9df8003ee0 con 0x7f9e001457f0 2024-03-29T11:40:41.148 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.140+0000 7f9de77fe640 1 -- 172.21.15.72:0/735612822 <== mon.2 v1:172.21.15.72:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f9df8084df0 con 0x7f9e001457f0 2024-03-29T11:40:41.148 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.140+0000 7f9de77fe640 10 client.4445.objecter ms_dispatch 0x7f9e00000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:41.148 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.140+0000 7f9de77fe640 3 client.4445.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:41.148 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.140+0000 7f9de77fe640 3 client.4445.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:41.148 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.140+0000 7f9de77fe640 20 client.4445.objecter dump_active .. 0 homeless 2024-03-29T11:40:41.148 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.140+0000 7f9dfeffd640 10 monclient: get_auth_request con 0x7f9dec06dbe0 auth_method 0 2024-03-29T11:40:41.148 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.141+0000 7f9de77fe640 10 client.4445.objecter ms_handle_connect 0x7f9dec06dbe0 2024-03-29T11:40:41.149 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.147+0000 7f9de77fe640 1 -- 172.21.15.72:0/735612822 <== mon.2 v1:172.21.15.72:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9df80506d0 con 0x7f9e001457f0 2024-03-29T11:40:41.149 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.147+0000 7f9de77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:41.149 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.147+0000 7f9de77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:41.451 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.449+0000 7f9e057b5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:41.451 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.449+0000 7f9e057b5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:41.452 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.449+0000 7f9e057b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:40:41.452 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.449+0000 7f9e057b5640 1 -- 172.21.15.72:0/735612822 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9de0005330 con 0x7f9e001457f0 2024-03-29T11:40:41.453 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:40:41.453 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:41.453 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.451+0000 7f9de77fe640 1 -- 172.21.15.72:0/735612822 <== mon.2 v1:172.21.15.72:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f9df8052600 con 0x7f9e001457f0 2024-03-29T11:40:41.453 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.451+0000 7f9de77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:41.454 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.451+0000 7f9de77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:41.456 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.453+0000 7f9e057b5640 10 librados: watch_flush enter 2024-03-29T11:40:41.456 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.453+0000 7f9e057b5640 10 librados: watch_flush exit 2024-03-29T11:40:41.456 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.453+0000 7f9e057b5640 20 client.4445.objecter shutdown clearing up homeless session... 2024-03-29T11:40:41.456 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.453+0000 7f9e057b5640 10 client.4445.objecter successfully canceled tick 2024-03-29T11:40:41.456 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.453+0000 7f9e057b5640 1 -- 172.21.15.72:0/735612822 >> v1:172.21.15.72:6816/1578866102 conn(0x7f9dec06dbe0 legacy=0x7f9dec0700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:41.456 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.453+0000 7f9e057b5640 10 monclient: shutdown 2024-03-29T11:40:41.456 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.453+0000 7f9e057b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:41.456 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.453+0000 7f9e057b5640 1 -- 172.21.15.72:0/735612822 >> v1:172.21.15.72:6790/0 conn(0x7f9e001457f0 legacy=0x7f9e00133890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:41.456 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.453+0000 7f9e057b5640 1 -- 172.21.15.72:0/735612822 shutdown_connections 2024-03-29T11:40:41.456 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.453+0000 7f9e057b5640 1 -- 172.21.15.72:0/735612822 >> 172.21.15.72:0/735612822 conn(0x7f9e00066a00 msgr2=0x7f9e000b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:41.456 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.453+0000 7f9e057b5640 1 -- 172.21.15.72:0/735612822 shutdown_connections 2024-03-29T11:40:41.456 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.453+0000 7f9e057b5640 1 -- 172.21.15.72:0/735612822 wait complete. 2024-03-29T11:40:41.456 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:41.454+0000 7f9e057b5640 1 librados: shutdown 2024-03-29T11:40:41.470 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:48.277 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:48.412 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.409+0000 7f43dbd44640 10 monclient: get_monmap_and_config 2024-03-29T11:40:48.412 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.409+0000 7f43dbd44640 10 monclient: build_initial_monmap 2024-03-29T11:40:48.412 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.409+0000 7f43dbd44640 10 monclient: monmap: 2024-03-29T11:40:48.412 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:40:48.412 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:40:48.412 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:40:48.410671+0000 2024-03-29T11:40:48.412 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:40:48.410671+0000 2024-03-29T11:40:48.412 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:48.412 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:48.412 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:40:48.412 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:40:48.412 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:40:48.412 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:48.413 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.410+0000 7f43dbd44640 1 Processor -- start 2024-03-29T11:40:48.413 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.410+0000 7f43dbd44640 1 -- start start 2024-03-29T11:40:48.413 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.410+0000 7f43dbd44640 10 monclient: init 2024-03-29T11:40:48.413 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.411+0000 7f43dbd44640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:48.414 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.411+0000 7f43dbd44640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:40:48.414 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.411+0000 7f43dbd44640 10 monclient(hunting): picked mon.noname-a con 0x7f43d41457f0 addr v1:172.21.15.72:6789/0 2024-03-29T11:40:48.414 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.411+0000 7f43dbd44640 10 monclient(hunting): picked mon.noname-c con 0x7f43d413f650 addr v1:172.21.15.89:6789/0 2024-03-29T11:40:48.414 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.412+0000 7f43dbd44640 10 monclient(hunting): picked mon.noname-b con 0x7f43d40a9d70 addr v1:172.21.15.72:6790/0 2024-03-29T11:40:48.414 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.412+0000 7f43dbd44640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d40aaef0 con 0x7f43d41457f0 2024-03-29T11:40:48.414 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.412+0000 7f43dbd44640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d4145bd0 con 0x7f43d413f650 2024-03-29T11:40:48.414 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.412+0000 7f43dbd44640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d4145eb0 con 0x7f43d40a9d70 2024-03-29T11:40:48.414 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.412+0000 7f43dbd44640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:48.414 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.412+0000 7f43dbd44640 10 monclient(hunting): authenticate will time out at 1207.700056s 2024-03-29T11:40:48.415 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.412+0000 7f43da2e1640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f43d40a9d70 0x7f43d414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:39850/0 (socket says 172.21.15.72:39850) 2024-03-29T11:40:48.415 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.412+0000 7f43da2e1640 1 -- 172.21.15.72:0/1355979759 learned_addr learned my addr 172.21.15.72:0/1355979759 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:40:48.415 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.412+0000 7f43d8ade640 1 -- 172.21.15.72:0/1355979759 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 695155118 0 0) 0x7f43d4145eb0 con 0x7f43d40a9d70 2024-03-29T11:40:48.415 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.412+0000 7f43d8ade640 10 monclient(hunting): _negotiate 2024-03-29T11:40:48.415 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.412+0000 7f43d8ade640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:48.415 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.412+0000 7f43d8ade640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:48.415 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.412+0000 7f43d8ade640 10 monclient(hunting): my global_id is 4448 2024-03-29T11:40:48.415 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.412+0000 7f43d8ade640 1 -- 172.21.15.72:0/1355979759 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43c0003520 con 0x7f43d40a9d70 2024-03-29T11:40:48.415 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.413+0000 7f43d8ade640 1 -- 172.21.15.72:0/1355979759 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 519796463 0 0) 0x7f43c0003520 con 0x7f43d40a9d70 2024-03-29T11:40:48.415 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.413+0000 7f43d8ade640 1 -- 172.21.15.72:0/1355979759 >> v1:172.21.15.89:6789/0 conn(0x7f43d413f650 legacy=0x7f43d40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:48.415 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.413+0000 7f43d8ade640 1 -- 172.21.15.72:0/1355979759 >> v1:172.21.15.72:6789/0 conn(0x7f43d41457f0 legacy=0x7f43d413ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:48.417 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.415+0000 7f43d8ade640 10 monclient: _finish_hunting 0 2024-03-29T11:40:48.418 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.415+0000 7f43d8ade640 1 monclient: found mon.noname-b 2024-03-29T11:40:48.418 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.415+0000 7f43d8ade640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:48.418 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.415+0000 7f43d8ade640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.72:6790/0 2024-03-29T11:40:48.418 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.415+0000 7f43d8ade640 1 -- 172.21.15.72:0/1355979759 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d4146340 con 0x7f43d40a9d70 2024-03-29T11:40:48.418 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.415+0000 7f43d8ade640 10 monclient: _finish_auth 0 2024-03-29T11:40:48.418 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.415+0000 7f43d8ade640 10 monclient: _check_auth_tickets 2024-03-29T11:40:48.418 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.415+0000 7f43d8ade640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:48.418 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.415+0000 7f43d8ade640 1 -- 172.21.15.72:0/1355979759 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f43cc001a10 con 0x7f43d40a9d70 2024-03-29T11:40:48.418 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.415+0000 7f43d8ade640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:48.418 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.415+0000 7f43d8ade640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:48.418 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.415+0000 7f43d8ade640 10 monclient: dump: 2024-03-29T11:40:48.419 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:40:48.419 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:40:48.419 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:48.419 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:48.419 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:48.419 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:48.419 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:40:48.419 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:40:48.419 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:40:48.419 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:48.419 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.415+0000 7f43dbd44640 5 monclient: authenticate success, global_id 4448 2024-03-29T11:40:48.419 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.415+0000 7f43dbd44640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:48.419 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.415+0000 7f43d8ade640 1 -- 172.21.15.72:0/1355979759 <== mon.2 v1:172.21.15.72:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43cc001be0 con 0x7f43d40a9d70 2024-03-29T11:40:48.419 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.415+0000 7f43d8ade640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:48.419 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.415+0000 7f43d8ade640 1 -- 172.21.15.72:0/1355979759 <== mon.2 v1:172.21.15.72:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f43cc002030 con 0x7f43d40a9d70 2024-03-29T11:40:48.420 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.415+0000 7f43dbd44640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:48.420 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.415+0000 7f43dbd44640 4 set_mon_vals no callback set 2024-03-29T11:40:48.420 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.415+0000 7f43d8ade640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:48.420 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.415+0000 7f43d8ade640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:48.420 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.415+0000 7f43d8ade640 10 monclient: dump: 2024-03-29T11:40:48.420 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:40:48.420 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:40:48.420 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:48.420 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:48.420 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:48.421 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:48.421 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:40:48.421 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:40:48.421 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:40:48.421 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:48.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.416+0000 7f43dbd44640 10 monclient: shutdown 2024-03-29T11:40:48.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.416+0000 7f43dbd44640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:48.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.416+0000 7f43dbd44640 1 -- 172.21.15.72:0/1355979759 >> v1:172.21.15.72:6790/0 conn(0x7f43d40a9d70 legacy=0x7f43d414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:48.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.416+0000 7f43dbd44640 1 -- 172.21.15.72:0/1355979759 shutdown_connections 2024-03-29T11:40:48.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.416+0000 7f43dbd44640 1 -- 172.21.15.72:0/1355979759 >> 172.21.15.72:0/1355979759 conn(0x7f43d4066a00 msgr2=0x7f43d4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:48.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.417+0000 7f43dbd44640 1 -- 172.21.15.72:0/1355979759 shutdown_connections 2024-03-29T11:40:48.430 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.417+0000 7f43dbd44640 1 -- 172.21.15.72:0/1355979759 wait complete. 2024-03-29T11:40:48.430 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.417+0000 7f43dbd44640 10 monclient: build_initial_monmap 2024-03-29T11:40:48.430 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.418+0000 7f43dbd44640 10 monclient: monmap: 2024-03-29T11:40:48.430 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:40:48.430 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:48.430 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:40:48.430 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:40:48.430 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:48.430 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:48.430 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:40:48.430 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:40:48.430 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:40:48.431 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:48.431 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.418+0000 7f43dbd44640 1 librados: starting msgr at 2024-03-29T11:40:48.431 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.418+0000 7f43dbd44640 1 librados: starting objecter 2024-03-29T11:40:48.431 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.418+0000 7f43dbd44640 1 Processor -- start 2024-03-29T11:40:48.431 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.418+0000 7f43dbd44640 1 -- start start 2024-03-29T11:40:48.431 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.418+0000 7f43dbd44640 1 librados: setting wanted keys 2024-03-29T11:40:48.431 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.418+0000 7f43dbd44640 1 librados: calling monclient init 2024-03-29T11:40:48.431 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.418+0000 7f43dbd44640 10 monclient: init 2024-03-29T11:40:48.431 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.419+0000 7f43dbd44640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:48.431 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.419+0000 7f43dbd44640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:40:48.431 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.419+0000 7f43dbd44640 10 monclient(hunting): picked mon.noname-a con 0x7f43d40a9d70 addr v1:172.21.15.72:6789/0 2024-03-29T11:40:48.432 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.419+0000 7f43dbd44640 10 monclient(hunting): picked mon.noname-c con 0x7f43d413f650 addr v1:172.21.15.72:6790/0 2024-03-29T11:40:48.432 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.419+0000 7f43dbd44640 10 monclient(hunting): picked mon.noname-b con 0x7f43d41457f0 addr v1:172.21.15.89:6789/0 2024-03-29T11:40:48.432 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.419+0000 7f43dbd44640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d4137e80 con 0x7f43d40a9d70 2024-03-29T11:40:48.432 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.419+0000 7f43dbd44640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d41dfa50 con 0x7f43d41457f0 2024-03-29T11:40:48.432 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.419+0000 7f43dbd44640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43d406f0d0 con 0x7f43d413f650 2024-03-29T11:40:48.432 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.419+0000 7f43dbd44640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:48.432 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.419+0000 7f43dbd44640 10 monclient(hunting): authenticate will time out at 1207.707209s 2024-03-29T11:40:48.432 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.419+0000 7f43d92df640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f43d413f650 0x7f43d41dec30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:40030/0 (socket says 172.21.15.72:40030) 2024-03-29T11:40:48.432 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.419+0000 7f43d92df640 1 -- 172.21.15.72:0/2958115707 learned_addr learned my addr 172.21.15.72:0/2958115707 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:40:48.433 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.419+0000 7f43ca7fc640 10 client.?.objecter ms_handle_connect 0x7f43d413f650 2024-03-29T11:40:48.433 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.419+0000 7f43ca7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:48.433 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.419+0000 7f43ca7fc640 1 -- 172.21.15.72:0/2958115707 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1269183088 0 0) 0x7f43d406f0d0 con 0x7f43d413f650 2024-03-29T11:40:48.433 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.419+0000 7f43ca7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:40:48.433 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.419+0000 7f43ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:48.433 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.419+0000 7f43ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:48.433 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.419+0000 7f43ca7fc640 10 monclient(hunting): my global_id is 4451 2024-03-29T11:40:48.433 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.419+0000 7f43ca7fc640 1 -- 172.21.15.72:0/2958115707 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43b8004eb0 con 0x7f43d413f650 2024-03-29T11:40:48.433 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.420+0000 7f43ca7fc640 1 -- 172.21.15.72:0/2958115707 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1981507795 0 0) 0x7f43b8004eb0 con 0x7f43d413f650 2024-03-29T11:40:48.433 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.420+0000 7f43ca7fc640 1 -- 172.21.15.72:0/2958115707 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43d406f0d0 con 0x7f43d413f650 2024-03-29T11:40:48.433 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.420+0000 7f43ca7fc640 1 -- 172.21.15.72:0/2958115707 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f43c4001a30 con 0x7f43d413f650 2024-03-29T11:40:48.433 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.420+0000 7f43ca7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:48.433 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.420+0000 7f43ca7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:48.433 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.420+0000 7f43ca7fc640 10 monclient(hunting): dump: 2024-03-29T11:40:48.433 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:40:48.434 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:40:48.434 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:48.434 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:48.434 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:48.434 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:48.434 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:40:48.434 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:40:48.434 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:40:48.434 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:48.434 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.420+0000 7f43ca7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:48.434 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.420+0000 7f43ca7fc640 1 -- 172.21.15.72:0/2958115707 <== mon.2 v1:172.21.15.72:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1098363386 0 0) 0x7f43d406f0d0 con 0x7f43d413f650 2024-03-29T11:40:48.434 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.420+0000 7f43ca7fc640 1 -- 172.21.15.72:0/2958115707 >> v1:172.21.15.89:6789/0 conn(0x7f43d41457f0 legacy=0x7f43d41df340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:48.434 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.420+0000 7f43ca7fc640 1 -- 172.21.15.72:0/2958115707 >> v1:172.21.15.72:6789/0 conn(0x7f43d40a9d70 legacy=0x7f43d4144a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:40:48.434 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.422+0000 7f43ca7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:40:48.435 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.422+0000 7f43ca7fc640 1 monclient: found mon.c 2024-03-29T11:40:48.435 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.423+0000 7f43ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:48.435 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.423+0000 7f43ca7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:40:48.435 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.423+0000 7f43ca7fc640 1 -- 172.21.15.72:0/2958115707 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43d406f3b0 con 0x7f43d413f650 2024-03-29T11:40:48.435 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.423+0000 7f43ca7fc640 10 monclient: _finish_auth 0 2024-03-29T11:40:48.435 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.423+0000 7f43ca7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:40:48.435 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.423+0000 7f43ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:48.435 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.423+0000 7f43dbd44640 5 monclient: authenticate success, global_id 4451 2024-03-29T11:40:48.435 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.423+0000 7f43dbd44640 10 monclient: _renew_subs 2024-03-29T11:40:48.435 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.423+0000 7f43dbd44640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:40:48.435 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.423+0000 7f43dbd44640 1 -- 172.21.15.72:0/2958115707 --> v1:172.21.15.72:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43d4137b10 con 0x7f43d413f650 2024-03-29T11:40:48.435 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.423+0000 7f43ca7fc640 1 -- 172.21.15.72:0/2958115707 <== mon.2 v1:172.21.15.72:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43c4001bb0 con 0x7f43d413f650 2024-03-29T11:40:48.435 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.423+0000 7f43ca7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:48.435 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.423+0000 7f43cbfff640 4 set_mon_vals no callback set 2024-03-29T11:40:48.435 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.423+0000 7f43dbd44640 10 client.4451.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:48.435 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.423+0000 7f43dbd44640 10 monclient: _renew_subs 2024-03-29T11:40:48.436 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.423+0000 7f43ca7fc640 1 -- 172.21.15.72:0/2958115707 <== mon.2 v1:172.21.15.72:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f43c4003a60 con 0x7f43d413f650 2024-03-29T11:40:48.436 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.423+0000 7f43dbd44640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:40:48.436 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.423+0000 7f43dbd44640 1 -- 172.21.15.72:0/2958115707 --> v1:172.21.15.72:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f43d406e0d0 con 0x7f43d413f650 2024-03-29T11:40:48.436 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.423+0000 7f43dbd44640 1 librados: init done 2024-03-29T11:40:48.436 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.423+0000 7f43ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:48.436 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.424+0000 7f43ca7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:48.436 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.424+0000 7f43ca7fc640 10 monclient: dump: 2024-03-29T11:40:48.436 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:40:48.436 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:40:48.436 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:48.436 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:48.436 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:48.436 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:48.436 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:40:48.436 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:40:48.436 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:40:48.437 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:48.437 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.425+0000 7f43ca7fc640 1 -- 172.21.15.72:0/2958115707 <== mon.2 v1:172.21.15.72:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f43c4003bc0 con 0x7f43d413f650 2024-03-29T11:40:48.437 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.425+0000 7f43d9ae0640 10 monclient: get_auth_request con 0x7f43b806dd20 auth_method 0 2024-03-29T11:40:48.437 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.425+0000 7f43ca7fc640 1 -- 172.21.15.72:0/2958115707 <== mon.2 v1:172.21.15.72:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f43c4084c00 con 0x7f43d413f650 2024-03-29T11:40:48.437 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.425+0000 7f43ca7fc640 10 client.4451.objecter ms_dispatch 0x7f43d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:48.437 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.425+0000 7f43ca7fc640 3 client.4451.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:48.437 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.425+0000 7f43ca7fc640 3 client.4451.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:48.437 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.425+0000 7f43ca7fc640 20 client.4451.objecter dump_active .. 0 homeless 2024-03-29T11:40:48.437 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.426+0000 7f43ca7fc640 10 client.4451.objecter ms_handle_connect 0x7f43b806dd20 2024-03-29T11:40:48.437 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.426+0000 7f43dbd44640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:48.437 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.426+0000 7f43dbd44640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:48.437 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.426+0000 7f43dbd44640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:40:48.437 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.426+0000 7f43dbd44640 1 -- 172.21.15.72:0/2958115707 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43d4075f10 con 0x7f43d413f650 2024-03-29T11:40:48.438 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.433+0000 7f43ca7fc640 1 -- 172.21.15.72:0/2958115707 <== mon.2 v1:172.21.15.72:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f43c40504e0 con 0x7f43d413f650 2024-03-29T11:40:48.438 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.433+0000 7f43ca7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:48.438 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.433+0000 7f43ca7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:48.745 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.743+0000 7f43dbd44640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:48.745 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.743+0000 7f43dbd44640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:48.745 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.743+0000 7f43dbd44640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:40:48.746 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.743+0000 7f43dbd44640 1 -- 172.21.15.72:0/2958115707 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43d40b3680 con 0x7f43d413f650 2024-03-29T11:40:48.746 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.744+0000 7f43ca7fc640 1 -- 172.21.15.72:0/2958115707 <== mon.2 v1:172.21.15.72:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f43c4052410 con 0x7f43d413f650 2024-03-29T11:40:48.746 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.744+0000 7f43ca7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:48.747 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.744+0000 7f43ca7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:48.747 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:40:48.747 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:48.750 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.747+0000 7f43dbd44640 10 librados: watch_flush enter 2024-03-29T11:40:48.750 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.747+0000 7f43dbd44640 10 librados: watch_flush exit 2024-03-29T11:40:48.750 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.747+0000 7f43dbd44640 20 client.4451.objecter shutdown clearing up homeless session... 2024-03-29T11:40:48.750 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.747+0000 7f43dbd44640 10 client.4451.objecter successfully canceled tick 2024-03-29T11:40:48.751 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.747+0000 7f43dbd44640 1 -- 172.21.15.72:0/2958115707 >> v1:172.21.15.72:6816/1578866102 conn(0x7f43b806dd20 legacy=0x7f43b80701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:48.751 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.747+0000 7f43dbd44640 10 monclient: shutdown 2024-03-29T11:40:48.751 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.747+0000 7f43dbd44640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:48.751 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.747+0000 7f43dbd44640 1 -- 172.21.15.72:0/2958115707 >> v1:172.21.15.72:6790/0 conn(0x7f43d413f650 legacy=0x7f43d41dec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:48.751 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.747+0000 7f43dbd44640 1 -- 172.21.15.72:0/2958115707 shutdown_connections 2024-03-29T11:40:48.751 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.747+0000 7f43dbd44640 1 -- 172.21.15.72:0/2958115707 >> 172.21.15.72:0/2958115707 conn(0x7f43d4066a00 msgr2=0x7f43d413aba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:48.751 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.748+0000 7f43dbd44640 1 -- 172.21.15.72:0/2958115707 shutdown_connections 2024-03-29T11:40:48.751 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.748+0000 7f43dbd44640 1 -- 172.21.15.72:0/2958115707 wait complete. 2024-03-29T11:40:48.751 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:48.748+0000 7f43dbd44640 1 librados: shutdown 2024-03-29T11:40:48.766 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:55.572 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:55.714 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.712+0000 7f978a8a6640 10 monclient: get_monmap_and_config 2024-03-29T11:40:55.714 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.712+0000 7f978a8a6640 10 monclient: build_initial_monmap 2024-03-29T11:40:55.714 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.712+0000 7f978a8a6640 10 monclient: monmap: 2024-03-29T11:40:55.714 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:40:55.714 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:40:55.714 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:40:55.713099+0000 2024-03-29T11:40:55.715 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:40:55.713099+0000 2024-03-29T11:40:55.715 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:55.715 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:55.715 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:40:55.715 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:40:55.715 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:40:55.715 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:55.716 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.714+0000 7f978a8a6640 1 Processor -- start 2024-03-29T11:40:55.716 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.714+0000 7f978a8a6640 1 -- start start 2024-03-29T11:40:55.716 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.714+0000 7f978a8a6640 10 monclient: init 2024-03-29T11:40:55.718 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.715+0000 7f978a8a6640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:55.718 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.715+0000 7f978a8a6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:40:55.718 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.715+0000 7f978a8a6640 10 monclient(hunting): picked mon.noname-b con 0x7f977c0389e0 addr v1:172.21.15.72:6790/0 2024-03-29T11:40:55.720 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.716+0000 7f978a8a6640 10 monclient(hunting): picked mon.noname-a con 0x7f977c038dc0 addr v1:172.21.15.72:6789/0 2024-03-29T11:40:55.720 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.716+0000 7f978a8a6640 10 monclient(hunting): picked mon.noname-c con 0x7f977c030660 addr v1:172.21.15.89:6789/0 2024-03-29T11:40:55.720 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.716+0000 7f978a8a6640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f977c0c54c0 con 0x7f977c038dc0 2024-03-29T11:40:55.720 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.716+0000 7f978a8a6640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f977c036430 con 0x7f977c030660 2024-03-29T11:40:55.720 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.716+0000 7f978a8a6640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f977c039840 con 0x7f977c0389e0 2024-03-29T11:40:55.720 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.716+0000 7f978a8a6640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:55.720 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.716+0000 7f978a8a6640 10 monclient(hunting): authenticate will time out at 1215.004506s 2024-03-29T11:40:55.720 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.718+0000 7f97898a4640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f977c0389e0 0x7f977c033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:56426/0 (socket says 172.21.15.72:56426) 2024-03-29T11:40:55.720 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.718+0000 7f97898a4640 1 -- 172.21.15.72:0/1725898588 learned_addr learned my addr 172.21.15.72:0/1725898588 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:40:55.721 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.718+0000 7f97888a2640 1 -- 172.21.15.72:0/1725898588 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2516886630 0 0) 0x7f977c0c54c0 con 0x7f977c038dc0 2024-03-29T11:40:55.721 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.718+0000 7f97888a2640 10 monclient(hunting): _negotiate 2024-03-29T11:40:55.721 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.718+0000 7f97888a2640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:55.721 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.718+0000 7f97888a2640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:55.721 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.718+0000 7f97888a2640 10 monclient(hunting): my global_id is 4431 2024-03-29T11:40:55.721 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.718+0000 7f97888a2640 1 -- 172.21.15.72:0/1725898588 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9770003520 con 0x7f977c038dc0 2024-03-29T11:40:55.721 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.719+0000 7f97888a2640 1 -- 172.21.15.72:0/1725898588 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 989442934 0 0) 0x7f977c039840 con 0x7f977c0389e0 2024-03-29T11:40:55.721 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.719+0000 7f97888a2640 10 monclient(hunting): _negotiate 2024-03-29T11:40:55.722 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.719+0000 7f97888a2640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:55.722 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.719+0000 7f97888a2640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:55.722 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.719+0000 7f97888a2640 10 monclient(hunting): my global_id is 4454 2024-03-29T11:40:55.722 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.719+0000 7f97888a2640 1 -- 172.21.15.72:0/1725898588 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f977c0c54c0 con 0x7f977c0389e0 2024-03-29T11:40:55.722 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.719+0000 7f97888a2640 1 -- 172.21.15.72:0/1725898588 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 703994268 0 0) 0x7f9770003520 con 0x7f977c038dc0 2024-03-29T11:40:55.722 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.719+0000 7f97888a2640 1 -- 172.21.15.72:0/1725898588 >> v1:172.21.15.72:6790/0 conn(0x7f977c0389e0 legacy=0x7f977c033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:55.722 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.719+0000 7f97888a2640 1 -- 172.21.15.72:0/1725898588 >> v1:172.21.15.89:6789/0 conn(0x7f977c030660 legacy=0x7f977c0d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:40:55.722 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.719+0000 7f97888a2640 10 monclient: _finish_hunting 0 2024-03-29T11:40:55.722 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.719+0000 7f97888a2640 1 monclient: found mon.noname-a 2024-03-29T11:40:55.722 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.719+0000 7f97888a2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:55.722 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.719+0000 7f97888a2640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.72:6789/0 2024-03-29T11:40:55.722 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.719+0000 7f97888a2640 1 -- 172.21.15.72:0/1725898588 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f977c035690 con 0x7f977c038dc0 2024-03-29T11:40:55.722 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.719+0000 7f97888a2640 10 monclient: _finish_auth 0 2024-03-29T11:40:55.722 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.719+0000 7f97888a2640 10 monclient: _check_auth_tickets 2024-03-29T11:40:55.723 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.719+0000 7f97888a2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:55.723 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.719+0000 7f97888a2640 1 -- 172.21.15.72:0/1725898588 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f978404f730 con 0x7f977c038dc0 2024-03-29T11:40:55.723 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.719+0000 7f97888a2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:55.723 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.719+0000 7f97888a2640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:40:55.723 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.719+0000 7f97888a2640 10 monclient: dump: 2024-03-29T11:40:55.723 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:40:55.723 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:40:55.723 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:55.723 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:55.723 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:55.723 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:55.723 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:40:55.723 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:40:55.723 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:40:55.723 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:55.723 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.719+0000 7f97888a2640 1 -- 172.21.15.72:0/1725898588 <== mon.0 v1:172.21.15.72:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9784075f10 con 0x7f977c038dc0 2024-03-29T11:40:55.724 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.719+0000 7f97888a2640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:55.724 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.719+0000 7f97888a2640 1 -- 172.21.15.72:0/1725898588 <== mon.0 v1:172.21.15.72:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f97840758b0 con 0x7f977c038dc0 2024-03-29T11:40:55.724 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.719+0000 7f97888a2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:55.724 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.719+0000 7f97888a2640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:40:55.724 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.719+0000 7f97888a2640 10 monclient: dump: 2024-03-29T11:40:55.724 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:40:55.724 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:40:55.724 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:55.724 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:55.724 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:55.724 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:55.724 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:40:55.724 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:40:55.724 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:40:55.724 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:55.726 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.721+0000 7f978a8a6640 5 monclient: authenticate success, global_id 4431 2024-03-29T11:40:55.726 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.721+0000 7f978a8a6640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:55.726 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.721+0000 7f978a8a6640 4 set_mon_vals no callback set 2024-03-29T11:40:55.726 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.722+0000 7f978a8a6640 10 monclient: shutdown 2024-03-29T11:40:55.726 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.722+0000 7f978a8a6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:55.726 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.722+0000 7f978a8a6640 1 -- 172.21.15.72:0/1725898588 >> v1:172.21.15.72:6789/0 conn(0x7f977c038dc0 legacy=0x7f977c0cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:55.726 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.722+0000 7f978a8a6640 1 -- 172.21.15.72:0/1725898588 shutdown_connections 2024-03-29T11:40:55.726 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.722+0000 7f978a8a6640 1 -- 172.21.15.72:0/1725898588 >> 172.21.15.72:0/1725898588 conn(0x7f977c005c00 msgr2=0x7f977c006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:55.726 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.722+0000 7f978a8a6640 1 -- 172.21.15.72:0/1725898588 shutdown_connections 2024-03-29T11:40:55.726 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.723+0000 7f978a8a6640 1 -- 172.21.15.72:0/1725898588 wait complete. 2024-03-29T11:40:55.729 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.723+0000 7f978a8a6640 10 monclient: build_initial_monmap 2024-03-29T11:40:55.729 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.723+0000 7f978a8a6640 10 monclient: monmap: 2024-03-29T11:40:55.729 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:40:55.729 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:55.729 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:40:55.729 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:40:55.729 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:55.729 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:55.729 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:40:55.729 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:40:55.729 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:40:55.729 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:55.729 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.724+0000 7f978a8a6640 1 librados: starting msgr at 2024-03-29T11:40:55.729 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.724+0000 7f978a8a6640 1 librados: starting objecter 2024-03-29T11:40:55.729 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.724+0000 7f978a8a6640 1 Processor -- start 2024-03-29T11:40:55.730 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.724+0000 7f978a8a6640 1 -- start start 2024-03-29T11:40:55.730 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.724+0000 7f978a8a6640 1 librados: setting wanted keys 2024-03-29T11:40:55.730 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.724+0000 7f978a8a6640 1 librados: calling monclient init 2024-03-29T11:40:55.730 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.724+0000 7f978a8a6640 10 monclient: init 2024-03-29T11:40:55.730 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.725+0000 7f978a8a6640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:55.730 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.725+0000 7f978a8a6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:40:55.736 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.727+0000 7f978a8a6640 10 monclient(hunting): picked mon.noname-b con 0x7f977c030660 addr v1:172.21.15.89:6789/0 2024-03-29T11:40:55.736 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.727+0000 7f978a8a6640 10 monclient(hunting): picked mon.noname-a con 0x7f977c0389e0 addr v1:172.21.15.72:6789/0 2024-03-29T11:40:55.737 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.727+0000 7f978a8a6640 10 monclient(hunting): picked mon.noname-c con 0x7f977c0c7440 addr v1:172.21.15.72:6790/0 2024-03-29T11:40:55.737 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.727+0000 7f978a8a6640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f977c0c0c60 con 0x7f977c0389e0 2024-03-29T11:40:55.737 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.727+0000 7f978a8a6640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f977c0c6d40 con 0x7f977c030660 2024-03-29T11:40:55.737 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.727+0000 7f978a8a6640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f977c0c7020 con 0x7f977c0c7440 2024-03-29T11:40:55.737 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.727+0000 7f978a8a6640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:55.737 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.727+0000 7f978a8a6640 10 monclient(hunting): authenticate will time out at 1215.015686s 2024-03-29T11:40:55.737 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.728+0000 7f978a0a5640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f977c0c7440 0x7f977c0c0550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:57352/0 (socket says 172.21.15.72:57352) 2024-03-29T11:40:55.737 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.728+0000 7f978a0a5640 1 -- 172.21.15.72:0/847991780 learned_addr learned my addr 172.21.15.72:0/847991780 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:40:55.737 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.728+0000 7f977a7fc640 10 client.?.objecter ms_handle_connect 0x7f977c0c7440 2024-03-29T11:40:55.737 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.728+0000 7f977a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:55.737 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.728+0000 7f977a7fc640 10 client.?.objecter ms_handle_connect 0x7f977c030660 2024-03-29T11:40:55.737 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.728+0000 7f977a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:55.737 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.728+0000 7f977a7fc640 1 -- 172.21.15.72:0/847991780 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3335959016 0 0) 0x7f977c0c6d40 con 0x7f977c030660 2024-03-29T11:40:55.737 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.728+0000 7f977a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:40:55.737 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.728+0000 7f977a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:55.738 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.728+0000 7f977a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:55.738 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.728+0000 7f977a7fc640 10 monclient(hunting): my global_id is 4399 2024-03-29T11:40:55.738 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.728+0000 7f977a7fc640 1 -- 172.21.15.72:0/847991780 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9768004eb0 con 0x7f977c030660 2024-03-29T11:40:55.738 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.728+0000 7f977a7fc640 1 -- 172.21.15.72:0/847991780 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2517312154 0 0) 0x7f977c0c7020 con 0x7f977c0c7440 2024-03-29T11:40:55.738 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.728+0000 7f977a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:40:55.738 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.728+0000 7f977a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:55.738 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.728+0000 7f977a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:55.738 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.728+0000 7f977a7fc640 10 monclient(hunting): my global_id is 4457 2024-03-29T11:40:55.738 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.728+0000 7f977a7fc640 1 -- 172.21.15.72:0/847991780 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f977c0c6d40 con 0x7f977c0c7440 2024-03-29T11:40:55.738 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.729+0000 7f977a7fc640 1 -- 172.21.15.72:0/847991780 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2083297957 0 0) 0x7f977c0c6d40 con 0x7f977c0c7440 2024-03-29T11:40:55.738 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.729+0000 7f977a7fc640 1 -- 172.21.15.72:0/847991780 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f977c0c7020 con 0x7f977c0c7440 2024-03-29T11:40:55.738 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.729+0000 7f977a7fc640 1 -- 172.21.15.72:0/847991780 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f9780003fa0 con 0x7f977c0c7440 2024-03-29T11:40:55.738 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.729+0000 7f977a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:55.738 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.729+0000 7f977a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:55.739 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.729+0000 7f977a7fc640 10 monclient(hunting): dump: 2024-03-29T11:40:55.739 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:40:55.739 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:40:55.739 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:55.739 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:55.739 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:55.739 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:55.739 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:40:55.739 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:40:55.739 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:40:55.739 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:55.739 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.729+0000 7f977a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:55.739 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.729+0000 7f977a7fc640 1 -- 172.21.15.72:0/847991780 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2675605064 0 0) 0x7f9768004eb0 con 0x7f977c030660 2024-03-29T11:40:55.739 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.729+0000 7f977a7fc640 1 -- 172.21.15.72:0/847991780 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f977c0c6d40 con 0x7f977c030660 2024-03-29T11:40:55.739 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.729+0000 7f977a7fc640 1 -- 172.21.15.72:0/847991780 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f9774001950 con 0x7f977c030660 2024-03-29T11:40:55.739 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.729+0000 7f977a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:55.740 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.729+0000 7f977a7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:40:55.740 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.729+0000 7f977a7fc640 10 monclient(hunting): dump: 2024-03-29T11:40:55.740 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:40:55.740 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:40:55.740 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:55.740 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:55.740 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:55.740 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:55.740 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:40:55.740 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:40:55.740 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:40:55.740 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:55.740 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.729+0000 7f977a7fc640 1 -- 172.21.15.72:0/847991780 <== mon.2 v1:172.21.15.72:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2948780319 0 0) 0x7f977c0c7020 con 0x7f977c0c7440 2024-03-29T11:40:55.740 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.729+0000 7f977a7fc640 1 -- 172.21.15.72:0/847991780 >> v1:172.21.15.89:6789/0 conn(0x7f977c030660 legacy=0x7f977c15cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:55.740 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.729+0000 7f977a7fc640 1 -- 172.21.15.72:0/847991780 >> v1:172.21.15.72:6789/0 conn(0x7f977c0389e0 legacy=0x7f977c0bfe40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:40:55.741 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.730+0000 7f977a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:40:55.741 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.730+0000 7f977a7fc640 1 monclient: found mon.c 2024-03-29T11:40:55.741 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.730+0000 7f977a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:55.741 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.730+0000 7f977a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:40:55.741 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.730+0000 7f977a7fc640 1 -- 172.21.15.72:0/847991780 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f977c15c6f0 con 0x7f977c0c7440 2024-03-29T11:40:55.741 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.730+0000 7f977a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:40:55.741 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.730+0000 7f977a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:40:55.741 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.730+0000 7f977a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:55.741 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.730+0000 7f978a8a6640 5 monclient: authenticate success, global_id 4457 2024-03-29T11:40:55.741 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.730+0000 7f978a8a6640 10 monclient: _renew_subs 2024-03-29T11:40:55.741 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.730+0000 7f978a8a6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:40:55.741 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.730+0000 7f978a8a6640 1 -- 172.21.15.72:0/847991780 --> v1:172.21.15.72:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f977c15d320 con 0x7f977c0c7440 2024-03-29T11:40:55.741 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.730+0000 7f978a8a6640 10 client.4457.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:55.741 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.730+0000 7f978a8a6640 10 monclient: _renew_subs 2024-03-29T11:40:55.741 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.730+0000 7f978a8a6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:40:55.741 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.730+0000 7f978a8a6640 1 -- 172.21.15.72:0/847991780 --> v1:172.21.15.72:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f977c0c5aa0 con 0x7f977c0c7440 2024-03-29T11:40:55.742 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.730+0000 7f977a7fc640 1 -- 172.21.15.72:0/847991780 <== mon.2 v1:172.21.15.72:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f97800044b0 con 0x7f977c0c7440 2024-03-29T11:40:55.742 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.730+0000 7f978a8a6640 1 librados: init done 2024-03-29T11:40:55.742 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.730+0000 7f977a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:55.742 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.730+0000 7f977a7fc640 1 -- 172.21.15.72:0/847991780 <== mon.2 v1:172.21.15.72:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f9780001070 con 0x7f977c0c7440 2024-03-29T11:40:55.742 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.730+0000 7f977a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:55.742 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.730+0000 7f977a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:55.742 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.730+0000 7f977a7fc640 10 monclient: dump: 2024-03-29T11:40:55.742 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:40:55.742 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:40:55.742 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:55.742 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:40:55.742 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:55.742 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:40:55.742 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:40:55.742 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:40:55.743 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:40:55.743 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:40:55.743 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.731+0000 7f977bfff640 4 set_mon_vals no callback set 2024-03-29T11:40:55.743 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.732+0000 7f9767fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:55.743 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.732+0000 7f9767fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:55.743 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.732+0000 7f9767fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:40:55.743 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.732+0000 7f9767fff640 1 -- 172.21.15.72:0/847991780 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9760005080 con 0x7f977c0c7440 2024-03-29T11:40:55.743 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.733+0000 7f977a7fc640 1 -- 172.21.15.72:0/847991780 <== mon.2 v1:172.21.15.72:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f97800011d0 con 0x7f977c0c7440 2024-03-29T11:40:55.743 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.734+0000 7f977a7fc640 1 -- 172.21.15.72:0/847991780 <== mon.2 v1:172.21.15.72:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f9780084980 con 0x7f977c0c7440 2024-03-29T11:40:55.743 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.734+0000 7f977a7fc640 10 client.4457.objecter ms_dispatch 0x7f9784000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:55.743 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.734+0000 7f977a7fc640 3 client.4457.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:55.743 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.734+0000 7f977a7fc640 3 client.4457.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:55.743 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.734+0000 7f97898a4640 10 monclient: get_auth_request con 0x7f976806dea0 auth_method 0 2024-03-29T11:40:55.744 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.734+0000 7f977a7fc640 20 client.4457.objecter dump_active .. 0 homeless 2024-03-29T11:40:55.744 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.734+0000 7f977a7fc640 10 client.4457.objecter ms_handle_connect 0x7f976806dea0 2024-03-29T11:40:55.744 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.742+0000 7f977a7fc640 1 -- 172.21.15.72:0/847991780 <== mon.2 v1:172.21.15.72:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9780050090 con 0x7f977c0c7440 2024-03-29T11:40:55.744 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.742+0000 7f977a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:55.745 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:55.742+0000 7f977a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:56.071 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:56.069+0000 7f9767fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:56.071 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:56.069+0000 7f9767fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:56.071 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:56.069+0000 7f9767fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:40:56.071 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:56.069+0000 7f9767fff640 1 -- 172.21.15.72:0/847991780 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9760005330 con 0x7f977c0c7440 2024-03-29T11:40:56.072 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:56.070+0000 7f977a7fc640 1 -- 172.21.15.72:0/847991780 <== mon.2 v1:172.21.15.72:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f9780052190 con 0x7f977c0c7440 2024-03-29T11:40:56.072 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:56.070+0000 7f977a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:56.072 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:56.070+0000 7f977a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:56.075 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:40:56.075 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:56.076 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:56.074+0000 7f9767fff640 10 librados: watch_flush enter 2024-03-29T11:40:56.076 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:56.074+0000 7f9767fff640 10 librados: watch_flush exit 2024-03-29T11:40:56.076 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:56.074+0000 7f9767fff640 20 client.4457.objecter shutdown clearing up homeless session... 2024-03-29T11:40:56.076 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:56.074+0000 7f9767fff640 10 client.4457.objecter successfully canceled tick 2024-03-29T11:40:56.077 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:56.075+0000 7f9767fff640 1 -- 172.21.15.72:0/847991780 >> v1:172.21.15.72:6816/1578866102 conn(0x7f976806dea0 legacy=0x7f9768070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:56.077 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:56.075+0000 7f9767fff640 10 monclient: shutdown 2024-03-29T11:40:56.077 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:56.075+0000 7f9767fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:56.077 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:56.075+0000 7f9767fff640 1 -- 172.21.15.72:0/847991780 >> v1:172.21.15.72:6790/0 conn(0x7f977c0c7440 legacy=0x7f977c0c0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:56.077 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:56.075+0000 7f9767fff640 1 -- 172.21.15.72:0/847991780 shutdown_connections 2024-03-29T11:40:56.078 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:56.075+0000 7f9767fff640 1 -- 172.21.15.72:0/847991780 >> 172.21.15.72:0/847991780 conn(0x7f977c005c00 msgr2=0x7f977c034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:56.078 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:56.075+0000 7f9767fff640 1 -- 172.21.15.72:0/847991780 shutdown_connections 2024-03-29T11:40:56.078 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:56.075+0000 7f9767fff640 1 -- 172.21.15.72:0/847991780 wait complete. 2024-03-29T11:40:56.078 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:40:56.075+0000 7f9767fff640 1 librados: shutdown 2024-03-29T11:40:56.094 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:02.900 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:03.052 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.049+0000 7f31f444a640 10 monclient: get_monmap_and_config 2024-03-29T11:41:03.052 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.049+0000 7f31f444a640 10 monclient: build_initial_monmap 2024-03-29T11:41:03.052 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.049+0000 7f31f444a640 10 monclient: monmap: 2024-03-29T11:41:03.052 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:41:03.052 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:41:03.052 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:41:03.050218+0000 2024-03-29T11:41:03.052 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:41:03.050218+0000 2024-03-29T11:41:03.052 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:03.052 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:03.052 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:41:03.052 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:41:03.052 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:41:03.053 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:03.053 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.049+0000 7f31f444a640 1 Processor -- start 2024-03-29T11:41:03.053 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.050+0000 7f31f444a640 1 -- start start 2024-03-29T11:41:03.053 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.050+0000 7f31f444a640 10 monclient: init 2024-03-29T11:41:03.053 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.051+0000 7f31f444a640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:03.054 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.051+0000 7f31f444a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:41:03.054 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.051+0000 7f31f444a640 10 monclient(hunting): picked mon.noname-c con 0x7f31ec0a9750 addr v1:172.21.15.89:6789/0 2024-03-29T11:41:03.057 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.051+0000 7f31f444a640 10 monclient(hunting): picked mon.noname-b con 0x7f31ec13e1e0 addr v1:172.21.15.72:6790/0 2024-03-29T11:41:03.057 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.055+0000 7f31f444a640 10 monclient(hunting): picked mon.noname-a con 0x7f31ec071c30 addr v1:172.21.15.72:6789/0 2024-03-29T11:41:03.058 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.055+0000 7f31f444a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31ec0ab300 con 0x7f31ec071c30 2024-03-29T11:41:03.058 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.055+0000 7f31f444a640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31ec0ac2f0 con 0x7f31ec0a9750 2024-03-29T11:41:03.058 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.055+0000 7f31f444a640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31ec0ad8d0 con 0x7f31ec13e1e0 2024-03-29T11:41:03.058 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.055+0000 7f31f444a640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:03.058 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.055+0000 7f31f444a640 10 monclient(hunting): authenticate will time out at 1222.343945s 2024-03-29T11:41:03.061 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.059+0000 7f31f21e6640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f31ec0a9750 0x7f31ec13dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:33533/0 (socket says 172.21.15.72:33533) 2024-03-29T11:41:03.064 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.059+0000 7f31f21e6640 1 -- 172.21.15.72:0/3818406521 learned_addr learned my addr 172.21.15.72:0/3818406521 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:41:03.065 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.063+0000 7f31f11e4640 1 -- 172.21.15.72:0/3818406521 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3957975101 0 0) 0x7f31ec0ac2f0 con 0x7f31ec0a9750 2024-03-29T11:41:03.065 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.063+0000 7f31f11e4640 10 monclient(hunting): _negotiate 2024-03-29T11:41:03.065 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.063+0000 7f31f11e4640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:03.065 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.063+0000 7f31f11e4640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:03.065 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.063+0000 7f31f11e4640 10 monclient(hunting): my global_id is 4402 2024-03-29T11:41:03.065 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.063+0000 7f31f11e4640 1 -- 172.21.15.72:0/3818406521 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31d8003520 con 0x7f31ec0a9750 2024-03-29T11:41:03.066 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.063+0000 7f31f11e4640 1 -- 172.21.15.72:0/3818406521 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2859662705 0 0) 0x7f31d8003520 con 0x7f31ec0a9750 2024-03-29T11:41:03.066 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.063+0000 7f31f11e4640 1 -- 172.21.15.72:0/3818406521 >> v1:172.21.15.72:6790/0 conn(0x7f31ec13e1e0 legacy=0x7f31ec0ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:03.066 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.063+0000 7f31f11e4640 1 -- 172.21.15.72:0/3818406521 >> v1:172.21.15.72:6789/0 conn(0x7f31ec071c30 legacy=0x7f31ec0ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:03.066 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.063+0000 7f31f11e4640 10 monclient: _finish_hunting 0 2024-03-29T11:41:03.066 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.063+0000 7f31f11e4640 1 monclient: found mon.noname-c 2024-03-29T11:41:03.066 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.063+0000 7f31f11e4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:03.066 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.063+0000 7f31f11e4640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.89:6789/0 2024-03-29T11:41:03.067 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.063+0000 7f31f11e4640 1 -- 172.21.15.72:0/3818406521 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31ec0ada60 con 0x7f31ec0a9750 2024-03-29T11:41:03.067 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.063+0000 7f31f11e4640 10 monclient: _finish_auth 0 2024-03-29T11:41:03.067 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.063+0000 7f31f11e4640 10 monclient: _check_auth_tickets 2024-03-29T11:41:03.067 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.063+0000 7f31f11e4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:03.067 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.063+0000 7f31f11e4640 1 -- 172.21.15.72:0/3818406521 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f31e80015c0 con 0x7f31ec0a9750 2024-03-29T11:41:03.067 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.063+0000 7f31f11e4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:03.067 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.063+0000 7f31f11e4640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:03.067 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.063+0000 7f31f11e4640 10 monclient: dump: 2024-03-29T11:41:03.067 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:41:03.067 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:41:03.067 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:03.067 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:03.067 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:03.067 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:03.067 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:41:03.068 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:41:03.068 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:41:03.068 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:03.068 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.064+0000 7f31f444a640 5 monclient: authenticate success, global_id 4402 2024-03-29T11:41:03.068 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.064+0000 7f31f444a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:03.068 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.064+0000 7f31f11e4640 1 -- 172.21.15.72:0/3818406521 <== mon.1 v1:172.21.15.89:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f31e80017e0 con 0x7f31ec0a9750 2024-03-29T11:41:03.068 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.064+0000 7f31f11e4640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:03.068 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.064+0000 7f31f11e4640 1 -- 172.21.15.72:0/3818406521 <== mon.1 v1:172.21.15.89:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f31e8001b70 con 0x7f31ec0a9750 2024-03-29T11:41:03.068 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.064+0000 7f31f11e4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:03.068 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.064+0000 7f31f11e4640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:41:03.068 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.064+0000 7f31f11e4640 10 monclient: dump: 2024-03-29T11:41:03.068 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:41:03.068 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:41:03.068 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:03.069 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:03.069 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:03.069 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:03.069 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:41:03.069 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:41:03.069 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:41:03.069 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:03.069 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.064+0000 7f31f444a640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:03.069 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.064+0000 7f31f444a640 4 set_mon_vals no callback set 2024-03-29T11:41:03.069 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.065+0000 7f31f444a640 10 monclient: shutdown 2024-03-29T11:41:03.070 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.065+0000 7f31f444a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:03.070 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.065+0000 7f31f444a640 1 -- 172.21.15.72:0/3818406521 >> v1:172.21.15.89:6789/0 conn(0x7f31ec0a9750 legacy=0x7f31ec13dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:03.070 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.065+0000 7f31f444a640 1 -- 172.21.15.72:0/3818406521 shutdown_connections 2024-03-29T11:41:03.070 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.065+0000 7f31f444a640 1 -- 172.21.15.72:0/3818406521 >> 172.21.15.72:0/3818406521 conn(0x7f31ec06b660 msgr2=0x7f31ec06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:03.070 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.068+0000 7f31f444a640 1 -- 172.21.15.72:0/3818406521 shutdown_connections 2024-03-29T11:41:03.071 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.069+0000 7f31f444a640 1 -- 172.21.15.72:0/3818406521 wait complete. 2024-03-29T11:41:03.073 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.070+0000 7f31f444a640 10 monclient: build_initial_monmap 2024-03-29T11:41:03.073 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.070+0000 7f31f444a640 10 monclient: monmap: 2024-03-29T11:41:03.073 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:41:03.073 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:03.073 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:41:03.073 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:41:03.073 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:03.073 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:03.073 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:41:03.074 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:41:03.074 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:41:03.074 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:03.074 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.070+0000 7f31f444a640 1 librados: starting msgr at 2024-03-29T11:41:03.074 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.070+0000 7f31f444a640 1 librados: starting objecter 2024-03-29T11:41:03.074 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.070+0000 7f31f444a640 1 Processor -- start 2024-03-29T11:41:03.074 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.071+0000 7f31f444a640 1 -- start start 2024-03-29T11:41:03.074 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.071+0000 7f31f444a640 1 librados: setting wanted keys 2024-03-29T11:41:03.074 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.071+0000 7f31f444a640 1 librados: calling monclient init 2024-03-29T11:41:03.074 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.071+0000 7f31f444a640 10 monclient: init 2024-03-29T11:41:03.075 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.072+0000 7f31f444a640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:03.075 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.072+0000 7f31f444a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:41:03.079 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.073+0000 7f31f444a640 10 monclient(hunting): picked mon.noname-a con 0x7f31ec071c30 addr v1:172.21.15.72:6789/0 2024-03-29T11:41:03.079 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.073+0000 7f31f444a640 10 monclient(hunting): picked mon.noname-b con 0x7f31ec0a9750 addr v1:172.21.15.89:6789/0 2024-03-29T11:41:03.079 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.073+0000 7f31f444a640 10 monclient(hunting): picked mon.noname-c con 0x7f31ec13e1e0 addr v1:172.21.15.72:6790/0 2024-03-29T11:41:03.079 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.073+0000 7f31f444a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31ec1daf60 con 0x7f31ec071c30 2024-03-29T11:41:03.079 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.073+0000 7f31f444a640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31ec1c8480 con 0x7f31ec0a9750 2024-03-29T11:41:03.079 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.073+0000 7f31f444a640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31ec1daa80 con 0x7f31ec13e1e0 2024-03-29T11:41:03.079 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.073+0000 7f31f444a640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:03.079 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.073+0000 7f31f444a640 10 monclient(hunting): authenticate will time out at 1222.361091s 2024-03-29T11:41:03.079 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.076+0000 7f31f19e5640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f31ec0a9750 0x7f31ec1d7580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:33679/0 (socket says 172.21.15.72:33679) 2024-03-29T11:41:03.079 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.076+0000 7f31f19e5640 1 -- 172.21.15.72:0/654993135 learned_addr learned my addr 172.21.15.72:0/654993135 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:41:03.084 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.076+0000 7f31e2ffd640 10 client.?.objecter ms_handle_connect 0x7f31ec0a9750 2024-03-29T11:41:03.084 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.076+0000 7f31e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:03.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.077+0000 7f31e2ffd640 1 -- 172.21.15.72:0/654993135 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2850527189 0 0) 0x7f31ec1c8480 con 0x7f31ec0a9750 2024-03-29T11:41:03.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.077+0000 7f31e2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:03.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.077+0000 7f31e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:03.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.077+0000 7f31e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:03.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.077+0000 7f31e2ffd640 10 monclient(hunting): my global_id is 4405 2024-03-29T11:41:03.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.077+0000 7f31e2ffd640 1 -- 172.21.15.72:0/654993135 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31d0004eb0 con 0x7f31ec0a9750 2024-03-29T11:41:03.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.078+0000 7f31e2ffd640 1 -- 172.21.15.72:0/654993135 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2047516625 0 0) 0x7f31d0004eb0 con 0x7f31ec0a9750 2024-03-29T11:41:03.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.078+0000 7f31e2ffd640 1 -- 172.21.15.72:0/654993135 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31ec1c8480 con 0x7f31ec0a9750 2024-03-29T11:41:03.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.078+0000 7f31e2ffd640 1 -- 172.21.15.72:0/654993135 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f31dc001990 con 0x7f31ec0a9750 2024-03-29T11:41:03.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.078+0000 7f31e2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:03.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.078+0000 7f31e2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:03.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.078+0000 7f31e2ffd640 10 monclient(hunting): dump: 2024-03-29T11:41:03.085 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:41:03.085 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:41:03.086 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:03.086 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:03.086 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:03.086 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:03.086 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:41:03.086 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:41:03.086 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:41:03.086 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:03.086 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.078+0000 7f31e2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:03.086 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.078+0000 7f31e2ffd640 1 -- 172.21.15.72:0/654993135 <== mon.1 v1:172.21.15.89:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 329110605 0 0) 0x7f31ec1c8480 con 0x7f31ec0a9750 2024-03-29T11:41:03.086 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.078+0000 7f31e2ffd640 1 -- 172.21.15.72:0/654993135 >> v1:172.21.15.72:6790/0 conn(0x7f31ec13e1e0 legacy=0x7f31ec1c9ec0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:41:03.086 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.080+0000 7f31e2ffd640 1 -- 172.21.15.72:0/654993135 >> v1:172.21.15.72:6789/0 conn(0x7f31ec071c30 legacy=0x7f31ec1db3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:03.086 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.080+0000 7f31e2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:41:03.086 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.080+0000 7f31e2ffd640 1 monclient: found mon.b 2024-03-29T11:41:03.087 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.080+0000 7f31e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:03.087 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.080+0000 7f31e2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:41:03.087 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.080+0000 7f31e2ffd640 1 -- 172.21.15.72:0/654993135 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31ec1dac10 con 0x7f31ec0a9750 2024-03-29T11:41:03.087 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.080+0000 7f31e2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:41:03.087 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.080+0000 7f31e2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:41:03.087 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.080+0000 7f31e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:03.087 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.080+0000 7f31f444a640 5 monclient: authenticate success, global_id 4405 2024-03-29T11:41:03.087 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.080+0000 7f31f444a640 10 monclient: _renew_subs 2024-03-29T11:41:03.087 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.080+0000 7f31f444a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:41:03.087 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.080+0000 7f31f444a640 1 -- 172.21.15.72:0/654993135 --> v1:172.21.15.89:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31ec139990 con 0x7f31ec0a9750 2024-03-29T11:41:03.087 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.081+0000 7f31f444a640 10 client.4405.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:03.087 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.081+0000 7f31f444a640 10 monclient: _renew_subs 2024-03-29T11:41:03.087 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.081+0000 7f31f444a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:41:03.087 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.081+0000 7f31f444a640 1 -- 172.21.15.72:0/654993135 --> v1:172.21.15.89:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f31ec0b27e0 con 0x7f31ec0a9750 2024-03-29T11:41:03.087 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.081+0000 7f31f444a640 1 librados: init done 2024-03-29T11:41:03.087 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.081+0000 7f31e2ffd640 10 client.4405.objecter ms_handle_connect 0x7f31ec071c30 2024-03-29T11:41:03.088 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.081+0000 7f31e2ffd640 10 client.4405.objecter resend_mon_ops 2024-03-29T11:41:03.088 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.084+0000 7f31e2ffd640 1 -- 172.21.15.72:0/654993135 <== mon.1 v1:172.21.15.89:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f31dc001c90 con 0x7f31ec0a9750 2024-03-29T11:41:03.089 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.084+0000 7f31e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:03.089 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.084+0000 7f31f444a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:03.089 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.084+0000 7f31f09e3640 4 set_mon_vals no callback set 2024-03-29T11:41:03.089 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.085+0000 7f31e2ffd640 1 -- 172.21.15.72:0/654993135 <== mon.1 v1:172.21.15.89:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f31dc003a60 con 0x7f31ec0a9750 2024-03-29T11:41:03.089 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.085+0000 7f31e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:03.089 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.085+0000 7f31e2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:41:03.089 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.085+0000 7f31e2ffd640 10 monclient: dump: 2024-03-29T11:41:03.089 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:41:03.089 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:41:03.089 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:03.089 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:03.089 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:03.089 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:03.089 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:41:03.089 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:41:03.089 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:41:03.089 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:03.090 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.085+0000 7f31f444a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:03.090 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.085+0000 7f31f444a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:41:03.090 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.085+0000 7f31f444a640 1 -- 172.21.15.72:0/654993135 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31ec0abb00 con 0x7f31ec0a9750 2024-03-29T11:41:03.090 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.087+0000 7f31e2ffd640 1 -- 172.21.15.72:0/654993135 <== mon.1 v1:172.21.15.89:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f31dc003c80 con 0x7f31ec0a9750 2024-03-29T11:41:03.091 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.088+0000 7f31e2ffd640 1 -- 172.21.15.72:0/654993135 <== mon.1 v1:172.21.15.89:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f31dc084b30 con 0x7f31ec0a9750 2024-03-29T11:41:03.091 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.088+0000 7f31e2ffd640 10 client.4405.objecter ms_dispatch 0x7f31ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:03.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.088+0000 7f31e2ffd640 3 client.4405.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:03.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.088+0000 7f31e2ffd640 3 client.4405.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:03.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.088+0000 7f31e2ffd640 20 client.4405.objecter dump_active .. 0 homeless 2024-03-29T11:41:03.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.089+0000 7f31f21e6640 10 monclient: get_auth_request con 0x7f31d0076470 auth_method 0 2024-03-29T11:41:03.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.090+0000 7f31e2ffd640 10 client.4405.objecter ms_handle_connect 0x7f31d0076470 2024-03-29T11:41:03.097 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.095+0000 7f31e2ffd640 1 -- 172.21.15.72:0/654993135 <== mon.1 v1:172.21.15.89:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f31dc050410 con 0x7f31ec0a9750 2024-03-29T11:41:03.098 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.095+0000 7f31e2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:03.098 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.095+0000 7f31e2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:03.420 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.418+0000 7f31f444a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:03.420 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.418+0000 7f31f444a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:03.421 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.418+0000 7f31f444a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:41:03.421 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.418+0000 7f31f444a640 1 -- 172.21.15.72:0/654993135 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31ec075f10 con 0x7f31ec0a9750 2024-03-29T11:41:03.422 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.419+0000 7f31e2ffd640 1 -- 172.21.15.72:0/654993135 <== mon.1 v1:172.21.15.89:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f31dc052340 con 0x7f31ec0a9750 2024-03-29T11:41:03.422 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.419+0000 7f31e2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:03.422 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.420+0000 7f31e2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:03.423 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:41:03.423 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:03.425 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.423+0000 7f31f444a640 10 librados: watch_flush enter 2024-03-29T11:41:03.426 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.423+0000 7f31f444a640 10 librados: watch_flush exit 2024-03-29T11:41:03.426 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.423+0000 7f31f444a640 20 client.4405.objecter shutdown clearing up homeless session... 2024-03-29T11:41:03.426 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.423+0000 7f31f444a640 10 client.4405.objecter successfully canceled tick 2024-03-29T11:41:03.426 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.423+0000 7f31f444a640 1 -- 172.21.15.72:0/654993135 >> v1:172.21.15.72:6816/1578866102 conn(0x7f31d0076470 legacy=0x7f31d0078930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:03.427 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.423+0000 7f31f444a640 10 monclient: shutdown 2024-03-29T11:41:03.427 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.423+0000 7f31f444a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:03.427 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.423+0000 7f31f444a640 1 -- 172.21.15.72:0/654993135 >> v1:172.21.15.89:6789/0 conn(0x7f31ec0a9750 legacy=0x7f31ec1d7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:03.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.425+0000 7f31f444a640 1 -- 172.21.15.72:0/654993135 shutdown_connections 2024-03-29T11:41:03.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.425+0000 7f31f444a640 1 -- 172.21.15.72:0/654993135 >> 172.21.15.72:0/654993135 conn(0x7f31ec06b660 msgr2=0x7f31ec06ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:03.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.425+0000 7f31f444a640 1 -- 172.21.15.72:0/654993135 shutdown_connections 2024-03-29T11:41:03.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.425+0000 7f31f444a640 1 -- 172.21.15.72:0/654993135 wait complete. 2024-03-29T11:41:03.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:03.425+0000 7f31f444a640 1 librados: shutdown 2024-03-29T11:41:03.445 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:10.251 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:10.405 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.402+0000 7f4e6140f640 10 monclient: get_monmap_and_config 2024-03-29T11:41:10.405 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.402+0000 7f4e6140f640 10 monclient: build_initial_monmap 2024-03-29T11:41:10.405 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.402+0000 7f4e6140f640 10 monclient: monmap: 2024-03-29T11:41:10.405 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:41:10.406 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:41:10.406 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:41:10.404313+0000 2024-03-29T11:41:10.406 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:41:10.404313+0000 2024-03-29T11:41:10.406 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:10.406 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:10.406 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:41:10.406 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:41:10.406 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:41:10.406 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:10.407 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.404+0000 7f4e6140f640 1 Processor -- start 2024-03-29T11:41:10.409 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.404+0000 7f4e6140f640 1 -- start start 2024-03-29T11:41:10.409 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.404+0000 7f4e6140f640 10 monclient: init 2024-03-29T11:41:10.410 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.405+0000 7f4e6140f640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:10.410 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.405+0000 7f4e6140f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:41:10.410 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.405+0000 7f4e6140f640 10 monclient(hunting): picked mon.noname-b con 0x7f4e5c1457f0 addr v1:172.21.15.72:6790/0 2024-03-29T11:41:10.410 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.405+0000 7f4e6140f640 10 monclient(hunting): picked mon.noname-a con 0x7f4e5c13f650 addr v1:172.21.15.72:6789/0 2024-03-29T11:41:10.410 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.405+0000 7f4e6140f640 10 monclient(hunting): picked mon.noname-c con 0x7f4e5c0a9d70 addr v1:172.21.15.89:6789/0 2024-03-29T11:41:10.410 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.405+0000 7f4e6140f640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e5c0aaef0 con 0x7f4e5c13f650 2024-03-29T11:41:10.410 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.405+0000 7f4e6140f640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e5c145bd0 con 0x7f4e5c0a9d70 2024-03-29T11:41:10.410 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.405+0000 7f4e6140f640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e5c145eb0 con 0x7f4e5c1457f0 2024-03-29T11:41:10.410 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.405+0000 7f4e6140f640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:10.410 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.405+0000 7f4e6140f640 10 monclient(hunting): authenticate will time out at 1229.694768s 2024-03-29T11:41:10.413 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.410+0000 7f4e5a7fc640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f4e5c13f650 0x7f4e5c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:57023/0 (socket says 172.21.15.72:57023) 2024-03-29T11:41:10.414 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.410+0000 7f4e5a7fc640 1 -- 172.21.15.72:0/539971861 learned_addr learned my addr 172.21.15.72:0/539971861 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:41:10.418 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.412+0000 7f4e59ffb640 1 -- 172.21.15.72:0/539971861 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 561942545 0 0) 0x7f4e5c0aaef0 con 0x7f4e5c13f650 2024-03-29T11:41:10.418 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.412+0000 7f4e59ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:41:10.418 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.412+0000 7f4e59ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:10.418 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.412+0000 7f4e59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:10.418 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.412+0000 7f4e59ffb640 10 monclient(hunting): my global_id is 4440 2024-03-29T11:41:10.418 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.412+0000 7f4e59ffb640 1 -- 172.21.15.72:0/539971861 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e44003520 con 0x7f4e5c13f650 2024-03-29T11:41:10.418 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.412+0000 7f4e59ffb640 1 -- 172.21.15.72:0/539971861 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2754885212 0 0) 0x7f4e5c145bd0 con 0x7f4e5c0a9d70 2024-03-29T11:41:10.418 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.413+0000 7f4e59ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:41:10.418 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.413+0000 7f4e59ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:10.418 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.413+0000 7f4e59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:10.418 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.413+0000 7f4e59ffb640 10 monclient(hunting): my global_id is 4408 2024-03-29T11:41:10.419 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.413+0000 7f4e59ffb640 1 -- 172.21.15.72:0/539971861 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e5c0aaef0 con 0x7f4e5c0a9d70 2024-03-29T11:41:10.419 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.413+0000 7f4e59ffb640 1 -- 172.21.15.72:0/539971861 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1576037687 0 0) 0x7f4e44003520 con 0x7f4e5c13f650 2024-03-29T11:41:10.419 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.413+0000 7f4e59ffb640 1 -- 172.21.15.72:0/539971861 >> v1:172.21.15.72:6790/0 conn(0x7f4e5c1457f0 legacy=0x7f4e5c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:10.419 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.413+0000 7f4e59ffb640 1 -- 172.21.15.72:0/539971861 >> v1:172.21.15.89:6789/0 conn(0x7f4e5c0a9d70 legacy=0x7f4e5c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:10.419 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.413+0000 7f4e59ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:41:10.419 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.413+0000 7f4e59ffb640 1 monclient: found mon.noname-a 2024-03-29T11:41:10.419 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.413+0000 7f4e59ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:10.419 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.413+0000 7f4e59ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.72:6789/0 2024-03-29T11:41:10.419 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.413+0000 7f4e59ffb640 1 -- 172.21.15.72:0/539971861 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e5c146340 con 0x7f4e5c13f650 2024-03-29T11:41:10.419 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.413+0000 7f4e59ffb640 10 monclient: _finish_auth 0 2024-03-29T11:41:10.419 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.413+0000 7f4e59ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:41:10.419 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.414+0000 7f4e59ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:10.419 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.414+0000 7f4e6140f640 5 monclient: authenticate success, global_id 4440 2024-03-29T11:41:10.419 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.414+0000 7f4e6140f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:10.419 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.414+0000 7f4e59ffb640 1 -- 172.21.15.72:0/539971861 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f4e4c001690 con 0x7f4e5c13f650 2024-03-29T11:41:10.420 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.414+0000 7f4e59ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:10.420 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.414+0000 7f4e59ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:41:10.420 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.414+0000 7f4e59ffb640 10 monclient: dump: 2024-03-29T11:41:10.420 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:41:10.420 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:41:10.420 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:10.420 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:10.420 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:10.420 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:10.420 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:41:10.420 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:41:10.420 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:41:10.420 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:10.420 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.414+0000 7f4e6140f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:10.420 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.414+0000 7f4e59ffb640 1 -- 172.21.15.72:0/539971861 <== mon.0 v1:172.21.15.72:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e4c0018b0 con 0x7f4e5c13f650 2024-03-29T11:41:10.421 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.414+0000 7f4e59ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:10.421 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.414+0000 7f4e6140f640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:10.422 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.414+0000 7f4e6140f640 4 set_mon_vals no callback set 2024-03-29T11:41:10.422 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.415+0000 7f4e6140f640 10 monclient: shutdown 2024-03-29T11:41:10.422 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.415+0000 7f4e6140f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:10.422 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.415+0000 7f4e6140f640 1 -- 172.21.15.72:0/539971861 >> v1:172.21.15.72:6789/0 conn(0x7f4e5c13f650 legacy=0x7f4e5c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:10.422 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.416+0000 7f4e6140f640 1 -- 172.21.15.72:0/539971861 shutdown_connections 2024-03-29T11:41:10.422 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.416+0000 7f4e6140f640 1 -- 172.21.15.72:0/539971861 >> 172.21.15.72:0/539971861 conn(0x7f4e5c066a00 msgr2=0x7f4e5c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:10.422 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.419+0000 7f4e6140f640 1 -- 172.21.15.72:0/539971861 shutdown_connections 2024-03-29T11:41:10.423 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.419+0000 7f4e6140f640 1 -- 172.21.15.72:0/539971861 wait complete. 2024-03-29T11:41:10.424 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.420+0000 7f4e6140f640 10 monclient: build_initial_monmap 2024-03-29T11:41:10.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.420+0000 7f4e6140f640 10 monclient: monmap: 2024-03-29T11:41:10.428 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:41:10.428 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:10.428 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:41:10.428 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:41:10.428 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:10.428 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:10.428 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:41:10.428 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:41:10.428 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:41:10.428 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:10.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.421+0000 7f4e6140f640 1 librados: starting msgr at 2024-03-29T11:41:10.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.421+0000 7f4e6140f640 1 librados: starting objecter 2024-03-29T11:41:10.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.421+0000 7f4e6140f640 1 Processor -- start 2024-03-29T11:41:10.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.424+0000 7f4e6140f640 1 -- start start 2024-03-29T11:41:10.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.424+0000 7f4e6140f640 1 librados: setting wanted keys 2024-03-29T11:41:10.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.424+0000 7f4e6140f640 1 librados: calling monclient init 2024-03-29T11:41:10.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.424+0000 7f4e6140f640 10 monclient: init 2024-03-29T11:41:10.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.425+0000 7f4e6140f640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:10.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.425+0000 7f4e6140f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:41:10.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.425+0000 7f4e6140f640 10 monclient(hunting): picked mon.noname-b con 0x7f4e5c0a9d70 addr v1:172.21.15.89:6789/0 2024-03-29T11:41:10.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.425+0000 7f4e6140f640 10 monclient(hunting): picked mon.noname-c con 0x7f4e5c13f650 addr v1:172.21.15.72:6790/0 2024-03-29T11:41:10.430 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.425+0000 7f4e6140f640 10 monclient(hunting): picked mon.noname-a con 0x7f4e5c1457f0 addr v1:172.21.15.72:6789/0 2024-03-29T11:41:10.430 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.426+0000 7f4e6140f640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e5c06f2f0 con 0x7f4e5c1457f0 2024-03-29T11:41:10.430 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.426+0000 7f4e6140f640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e5c06f560 con 0x7f4e5c0a9d70 2024-03-29T11:41:10.430 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.426+0000 7f4e6140f640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e5c142080 con 0x7f4e5c13f650 2024-03-29T11:41:10.430 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.426+0000 7f4e6140f640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:10.430 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.426+0000 7f4e6140f640 10 monclient(hunting): authenticate will time out at 1229.715033s 2024-03-29T11:41:10.432 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.429+0000 7f4e5a7fc640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f4e5c13f650 0x7f4e5c140a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:56281/0 (socket says 172.21.15.72:56281) 2024-03-29T11:41:10.432 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.429+0000 7f4e5a7fc640 1 -- 172.21.15.72:0/1445421567 learned_addr learned my addr 172.21.15.72:0/1445421567 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:41:10.436 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.433+0000 7f4e437fe640 10 client.?.objecter ms_handle_connect 0x7f4e5c13f650 2024-03-29T11:41:10.436 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.433+0000 7f4e437fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:10.438 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.435+0000 7f4e437fe640 10 client.?.objecter ms_handle_connect 0x7f4e5c0a9d70 2024-03-29T11:41:10.438 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.435+0000 7f4e437fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:10.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.436+0000 7f4e437fe640 10 client.?.objecter ms_handle_connect 0x7f4e5c1457f0 2024-03-29T11:41:10.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.436+0000 7f4e437fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:10.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.436+0000 7f4e437fe640 1 -- 172.21.15.72:0/1445421567 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1827974242 0 0) 0x7f4e5c06f2f0 con 0x7f4e5c1457f0 2024-03-29T11:41:10.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.436+0000 7f4e437fe640 10 monclient(hunting): _negotiate 2024-03-29T11:41:10.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.436+0000 7f4e437fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:10.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.436+0000 7f4e437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:10.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.436+0000 7f4e437fe640 10 monclient(hunting): my global_id is 4443 2024-03-29T11:41:10.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.436+0000 7f4e437fe640 1 -- 172.21.15.72:0/1445421567 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e48004eb0 con 0x7f4e5c1457f0 2024-03-29T11:41:10.442 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.436+0000 7f4e437fe640 1 -- 172.21.15.72:0/1445421567 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 214832867 0 0) 0x7f4e48004eb0 con 0x7f4e5c1457f0 2024-03-29T11:41:10.442 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.436+0000 7f4e437fe640 1 -- 172.21.15.72:0/1445421567 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e5c06f2f0 con 0x7f4e5c1457f0 2024-03-29T11:41:10.442 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.436+0000 7f4e437fe640 1 -- 172.21.15.72:0/1445421567 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f4e54001c70 con 0x7f4e5c1457f0 2024-03-29T11:41:10.442 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.436+0000 7f4e437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:10.442 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.436+0000 7f4e437fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:41:10.443 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.436+0000 7f4e437fe640 10 monclient(hunting): dump: 2024-03-29T11:41:10.443 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:41:10.443 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:41:10.443 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:10.443 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:10.443 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:10.443 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:10.443 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:41:10.443 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:41:10.443 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:41:10.443 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:10.443 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.436+0000 7f4e437fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:10.443 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.436+0000 7f4e437fe640 1 -- 172.21.15.72:0/1445421567 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1607263354 0 0) 0x7f4e5c142080 con 0x7f4e5c13f650 2024-03-29T11:41:10.443 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.437+0000 7f4e437fe640 10 monclient(hunting): _negotiate 2024-03-29T11:41:10.444 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.437+0000 7f4e437fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:10.444 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.437+0000 7f4e437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:10.444 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.437+0000 7f4e437fe640 10 monclient(hunting): my global_id is 4466 2024-03-29T11:41:10.444 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.437+0000 7f4e437fe640 1 -- 172.21.15.72:0/1445421567 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e48004eb0 con 0x7f4e5c13f650 2024-03-29T11:41:10.444 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.437+0000 7f4e437fe640 1 -- 172.21.15.72:0/1445421567 <== mon.0 v1:172.21.15.72:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 917487427 0 0) 0x7f4e5c06f2f0 con 0x7f4e5c1457f0 2024-03-29T11:41:10.444 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.437+0000 7f4e437fe640 1 -- 172.21.15.72:0/1445421567 >> v1:172.21.15.72:6790/0 conn(0x7f4e5c13f650 legacy=0x7f4e5c140a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:10.444 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.437+0000 7f4e437fe640 1 -- 172.21.15.72:0/1445421567 >> v1:172.21.15.89:6789/0 conn(0x7f4e5c0a9d70 legacy=0x7f4e5c140360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:10.444 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.437+0000 7f4e437fe640 10 monclient: _finish_hunting 0 2024-03-29T11:41:10.444 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.437+0000 7f4e437fe640 1 monclient: found mon.a 2024-03-29T11:41:10.444 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.437+0000 7f4e437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:10.444 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.437+0000 7f4e437fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:41:10.444 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.437+0000 7f4e437fe640 1 -- 172.21.15.72:0/1445421567 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e5c142ae0 con 0x7f4e5c1457f0 2024-03-29T11:41:10.444 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.437+0000 7f4e437fe640 10 monclient: _finish_auth 0 2024-03-29T11:41:10.444 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.437+0000 7f4e437fe640 10 monclient: _check_auth_tickets 2024-03-29T11:41:10.444 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.437+0000 7f4e437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:10.445 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.437+0000 7f4e6140f640 5 monclient: authenticate success, global_id 4443 2024-03-29T11:41:10.445 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.437+0000 7f4e6140f640 10 monclient: _renew_subs 2024-03-29T11:41:10.445 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.437+0000 7f4e6140f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:41:10.445 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.437+0000 7f4e6140f640 1 -- 172.21.15.72:0/1445421567 --> v1:172.21.15.72:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e5c142d90 con 0x7f4e5c1457f0 2024-03-29T11:41:10.445 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.437+0000 7f4e6140f640 10 client.4443.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:10.445 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.437+0000 7f4e6140f640 10 monclient: _renew_subs 2024-03-29T11:41:10.445 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.437+0000 7f4e6140f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:41:10.445 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.437+0000 7f4e6140f640 1 -- 172.21.15.72:0/1445421567 --> v1:172.21.15.72:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e5c06e2e0 con 0x7f4e5c1457f0 2024-03-29T11:41:10.445 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.437+0000 7f4e6140f640 1 librados: init done 2024-03-29T11:41:10.445 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.438+0000 7f4e437fe640 1 -- 172.21.15.72:0/1445421567 <== mon.0 v1:172.21.15.72:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e54001dd0 con 0x7f4e5c1457f0 2024-03-29T11:41:10.445 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.438+0000 7f4e437fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:10.445 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.438+0000 7f4e437fe640 1 -- 172.21.15.72:0/1445421567 <== mon.0 v1:172.21.15.72:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f4e54003c40 con 0x7f4e5c1457f0 2024-03-29T11:41:10.445 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.438+0000 7f4e437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:10.445 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.438+0000 7f4e437fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:10.445 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.438+0000 7f4e58ff9640 4 set_mon_vals no callback set 2024-03-29T11:41:10.447 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.440+0000 7f4e437fe640 10 monclient: dump: 2024-03-29T11:41:10.447 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:41:10.447 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:41:10.447 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:10.447 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:10.447 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:10.447 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:10.447 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:41:10.447 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:41:10.448 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:41:10.448 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:10.448 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.440+0000 7f4e437fe640 1 -- 172.21.15.72:0/1445421567 <== mon.0 v1:172.21.15.72:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f4e54004e90 con 0x7f4e5c1457f0 2024-03-29T11:41:10.448 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.441+0000 7f4e437fe640 1 -- 172.21.15.72:0/1445421567 <== mon.0 v1:172.21.15.72:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f4e54083c80 con 0x7f4e5c1457f0 2024-03-29T11:41:10.448 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.441+0000 7f4e5affd640 10 monclient: get_auth_request con 0x7f4e4806dd60 auth_method 0 2024-03-29T11:41:10.448 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.441+0000 7f4e437fe640 10 client.4443.objecter ms_dispatch 0x7f4e5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:10.448 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.442+0000 7f4e437fe640 3 client.4443.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:10.448 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.442+0000 7f4e437fe640 3 client.4443.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:10.448 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.442+0000 7f4e437fe640 20 client.4443.objecter dump_active .. 0 homeless 2024-03-29T11:41:10.448 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.442+0000 7f4e6140f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:10.448 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.442+0000 7f4e6140f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:10.448 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.442+0000 7f4e6140f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:41:10.448 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.442+0000 7f4e6140f640 1 -- 172.21.15.72:0/1445421567 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e3c005080 con 0x7f4e5c1457f0 2024-03-29T11:41:10.448 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.442+0000 7f4e437fe640 10 client.4443.objecter ms_handle_connect 0x7f4e4806dd60 2024-03-29T11:41:10.459 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.456+0000 7f4e437fe640 1 -- 172.21.15.72:0/1445421567 <== mon.0 v1:172.21.15.72:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4e54083e60 con 0x7f4e5c1457f0 2024-03-29T11:41:10.459 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.456+0000 7f4e437fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:10.459 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.456+0000 7f4e437fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:10.771 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.767+0000 7f4e6140f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:10.771 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.767+0000 7f4e6140f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:10.771 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.767+0000 7f4e6140f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:41:10.771 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.767+0000 7f4e6140f640 1 -- 172.21.15.72:0/1445421567 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e3c005330 con 0x7f4e5c1457f0 2024-03-29T11:41:10.772 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.769+0000 7f4e437fe640 1 -- 172.21.15.72:0/1445421567 <== mon.0 v1:172.21.15.72:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f4e54053120 con 0x7f4e5c1457f0 2024-03-29T11:41:10.773 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.769+0000 7f4e437fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:10.773 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.769+0000 7f4e437fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:10.775 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:41:10.775 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:10.775 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.772+0000 7f4e6140f640 10 librados: watch_flush enter 2024-03-29T11:41:10.776 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.772+0000 7f4e6140f640 10 librados: watch_flush exit 2024-03-29T11:41:10.776 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.772+0000 7f4e6140f640 20 client.4443.objecter shutdown clearing up homeless session... 2024-03-29T11:41:10.776 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.772+0000 7f4e6140f640 10 client.4443.objecter successfully canceled tick 2024-03-29T11:41:10.777 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.774+0000 7f4e6140f640 1 -- 172.21.15.72:0/1445421567 >> v1:172.21.15.72:6816/1578866102 conn(0x7f4e4806dd60 legacy=0x7f4e48070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:10.777 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.774+0000 7f4e6140f640 10 monclient: shutdown 2024-03-29T11:41:10.777 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.774+0000 7f4e6140f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:10.777 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.774+0000 7f4e6140f640 1 -- 172.21.15.72:0/1445421567 >> v1:172.21.15.72:6789/0 conn(0x7f4e5c1457f0 legacy=0x7f4e5c1eeab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:10.779 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.775+0000 7f4e6140f640 1 -- 172.21.15.72:0/1445421567 shutdown_connections 2024-03-29T11:41:10.779 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.775+0000 7f4e6140f640 1 -- 172.21.15.72:0/1445421567 >> 172.21.15.72:0/1445421567 conn(0x7f4e5c066a00 msgr2=0x7f4e5c0b3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:10.779 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.776+0000 7f4e6140f640 1 -- 172.21.15.72:0/1445421567 shutdown_connections 2024-03-29T11:41:10.779 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.776+0000 7f4e6140f640 1 -- 172.21.15.72:0/1445421567 wait complete. 2024-03-29T11:41:10.779 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:10.776+0000 7f4e6140f640 1 librados: shutdown 2024-03-29T11:41:10.799 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:17.606 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:17.758 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.756+0000 7fc8d8a8f640 10 monclient: get_monmap_and_config 2024-03-29T11:41:17.759 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.756+0000 7fc8d8a8f640 10 monclient: build_initial_monmap 2024-03-29T11:41:17.759 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.756+0000 7fc8d8a8f640 10 monclient: monmap: 2024-03-29T11:41:17.759 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:41:17.759 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:41:17.759 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:41:17.757178+0000 2024-03-29T11:41:17.759 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:41:17.757178+0000 2024-03-29T11:41:17.759 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:17.759 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:17.759 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:41:17.759 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:41:17.759 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:41:17.759 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:17.764 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.757+0000 7fc8d8a8f640 1 Processor -- start 2024-03-29T11:41:17.764 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.757+0000 7fc8d8a8f640 1 -- start start 2024-03-29T11:41:17.764 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.757+0000 7fc8d8a8f640 10 monclient: init 2024-03-29T11:41:17.764 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.758+0000 7fc8d8a8f640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:17.764 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.758+0000 7fc8d8a8f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:41:17.764 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.758+0000 7fc8d8a8f640 10 monclient(hunting): picked mon.noname-a con 0x7fc8d40b0770 addr v1:172.21.15.72:6789/0 2024-03-29T11:41:17.764 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.758+0000 7fc8d8a8f640 10 monclient(hunting): picked mon.noname-c con 0x7fc8d414a700 addr v1:172.21.15.89:6789/0 2024-03-29T11:41:17.764 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.758+0000 7fc8d8a8f640 10 monclient(hunting): picked mon.noname-b con 0x7fc8d40ac5a0 addr v1:172.21.15.72:6790/0 2024-03-29T11:41:17.764 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.758+0000 7fc8d8a8f640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8d40aba70 con 0x7fc8d40b0770 2024-03-29T11:41:17.764 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.758+0000 7fc8d8a8f640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8d40aecb0 con 0x7fc8d414a700 2024-03-29T11:41:17.764 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.758+0000 7fc8d8a8f640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8d40af150 con 0x7fc8d40ac5a0 2024-03-29T11:41:17.764 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.758+0000 7fc8d8a8f640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:17.765 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.758+0000 7fc8d8a8f640 10 monclient(hunting): authenticate will time out at 1237.046830s 2024-03-29T11:41:17.765 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.763+0000 7fc8d2d9d640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7fc8d40ac5a0 0x7fc8d4151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:55307/0 (socket says 172.21.15.72:55307) 2024-03-29T11:41:17.765 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.763+0000 7fc8d1d9b640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc8d414a700 0x7fc8d40abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:48319/0 (socket says 172.21.15.72:48319) 2024-03-29T11:41:17.766 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.763+0000 7fc8d2d9d640 1 -- 172.21.15.72:0/2571377419 learned_addr learned my addr 172.21.15.72:0/2571377419 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:41:17.768 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.765+0000 7fc8d159a640 1 -- 172.21.15.72:0/2571377419 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2207039254 0 0) 0x7fc8d40aecb0 con 0x7fc8d414a700 2024-03-29T11:41:17.768 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.765+0000 7fc8d159a640 10 monclient(hunting): _negotiate 2024-03-29T11:41:17.768 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.765+0000 7fc8d159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:17.769 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.765+0000 7fc8d159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:17.769 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.765+0000 7fc8d159a640 10 monclient(hunting): my global_id is 4414 2024-03-29T11:41:17.769 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.765+0000 7fc8d159a640 1 -- 172.21.15.72:0/2571377419 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8bc003520 con 0x7fc8d414a700 2024-03-29T11:41:17.769 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.765+0000 7fc8d159a640 1 -- 172.21.15.72:0/2571377419 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2310621201 0 0) 0x7fc8d40af150 con 0x7fc8d40ac5a0 2024-03-29T11:41:17.769 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.765+0000 7fc8d159a640 10 monclient(hunting): _negotiate 2024-03-29T11:41:17.769 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.765+0000 7fc8d159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:17.769 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.765+0000 7fc8d159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:17.769 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.765+0000 7fc8d159a640 10 monclient(hunting): my global_id is 4469 2024-03-29T11:41:17.769 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.765+0000 7fc8d159a640 1 -- 172.21.15.72:0/2571377419 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8d40aecb0 con 0x7fc8d40ac5a0 2024-03-29T11:41:17.769 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.765+0000 7fc8d159a640 1 -- 172.21.15.72:0/2571377419 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 124097145 0 0) 0x7fc8d40aba70 con 0x7fc8d40b0770 2024-03-29T11:41:17.769 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.765+0000 7fc8d159a640 10 monclient(hunting): _negotiate 2024-03-29T11:41:17.769 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.765+0000 7fc8d159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:17.770 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.765+0000 7fc8d159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:17.770 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.765+0000 7fc8d159a640 10 monclient(hunting): my global_id is 4446 2024-03-29T11:41:17.770 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.765+0000 7fc8d159a640 1 -- 172.21.15.72:0/2571377419 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8d40af150 con 0x7fc8d40b0770 2024-03-29T11:41:17.770 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.765+0000 7fc8d159a640 1 -- 172.21.15.72:0/2571377419 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1373375806 0 0) 0x7fc8d40af150 con 0x7fc8d40b0770 2024-03-29T11:41:17.770 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.765+0000 7fc8d159a640 1 -- 172.21.15.72:0/2571377419 >> v1:172.21.15.72:6790/0 conn(0x7fc8d40ac5a0 legacy=0x7fc8d4151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:17.770 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.765+0000 7fc8d159a640 1 -- 172.21.15.72:0/2571377419 >> v1:172.21.15.89:6789/0 conn(0x7fc8d414a700 legacy=0x7fc8d40abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:17.770 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.765+0000 7fc8d159a640 10 monclient: _finish_hunting 0 2024-03-29T11:41:17.770 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.765+0000 7fc8d159a640 1 monclient: found mon.noname-a 2024-03-29T11:41:17.770 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.765+0000 7fc8d159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:17.770 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.765+0000 7fc8d159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.72:6789/0 2024-03-29T11:41:17.770 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.765+0000 7fc8d159a640 1 -- 172.21.15.72:0/2571377419 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d40ac9d0 con 0x7fc8d40b0770 2024-03-29T11:41:17.770 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.765+0000 7fc8d159a640 10 monclient: _finish_auth 0 2024-03-29T11:41:17.770 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.765+0000 7fc8d159a640 10 monclient: _check_auth_tickets 2024-03-29T11:41:17.771 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.765+0000 7fc8d159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:17.771 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.765+0000 7fc8d159a640 1 -- 172.21.15.72:0/2571377419 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fc8cc0017e0 con 0x7fc8d40b0770 2024-03-29T11:41:17.771 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.765+0000 7fc8d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:17.771 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.765+0000 7fc8d159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:41:17.771 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.765+0000 7fc8d159a640 10 monclient: dump: 2024-03-29T11:41:17.771 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:41:17.771 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:41:17.771 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:17.771 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:17.771 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:17.771 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:17.771 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:41:17.771 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:41:17.772 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:41:17.772 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:17.772 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.766+0000 7fc8d159a640 1 -- 172.21.15.72:0/2571377419 <== mon.0 v1:172.21.15.72:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8cc001940 con 0x7fc8d40b0770 2024-03-29T11:41:17.772 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.766+0000 7fc8d159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:17.772 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.766+0000 7fc8d159a640 1 -- 172.21.15.72:0/2571377419 <== mon.0 v1:172.21.15.72:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fc8cc001d50 con 0x7fc8d40b0770 2024-03-29T11:41:17.772 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.766+0000 7fc8d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:17.772 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.766+0000 7fc8d159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:17.772 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.766+0000 7fc8d159a640 10 monclient: dump: 2024-03-29T11:41:17.772 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:41:17.772 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:41:17.772 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:17.772 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:17.772 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:17.772 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:17.773 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:41:17.773 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:41:17.773 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:41:17.773 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:17.773 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.766+0000 7fc8d8a8f640 5 monclient: authenticate success, global_id 4446 2024-03-29T11:41:17.773 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.766+0000 7fc8d8a8f640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:17.773 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.766+0000 7fc8d8a8f640 4 set_mon_vals no callback set 2024-03-29T11:41:17.774 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.767+0000 7fc8d8a8f640 10 monclient: shutdown 2024-03-29T11:41:17.774 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.767+0000 7fc8d8a8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:17.774 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.767+0000 7fc8d8a8f640 1 -- 172.21.15.72:0/2571377419 >> v1:172.21.15.72:6789/0 conn(0x7fc8d40b0770 legacy=0x7fc8d4149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:17.774 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.771+0000 7fc8d8a8f640 1 -- 172.21.15.72:0/2571377419 shutdown_connections 2024-03-29T11:41:17.774 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.771+0000 7fc8d8a8f640 1 -- 172.21.15.72:0/2571377419 >> 172.21.15.72:0/2571377419 conn(0x7fc8d406b660 msgr2=0x7fc8d406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:17.774 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.771+0000 7fc8d8a8f640 1 -- 172.21.15.72:0/2571377419 shutdown_connections 2024-03-29T11:41:17.774 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.771+0000 7fc8d8a8f640 1 -- 172.21.15.72:0/2571377419 wait complete. 2024-03-29T11:41:17.775 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.773+0000 7fc8d8a8f640 10 monclient: build_initial_monmap 2024-03-29T11:41:17.775 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.773+0000 7fc8d8a8f640 10 monclient: monmap: 2024-03-29T11:41:17.775 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:41:17.775 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:17.775 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:41:17.775 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:41:17.775 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:17.775 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:17.775 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:41:17.776 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:41:17.776 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:41:17.776 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:17.777 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.773+0000 7fc8d8a8f640 1 librados: starting msgr at 2024-03-29T11:41:17.777 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.773+0000 7fc8d8a8f640 1 librados: starting objecter 2024-03-29T11:41:17.777 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.773+0000 7fc8d8a8f640 1 Processor -- start 2024-03-29T11:41:17.777 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.773+0000 7fc8d8a8f640 1 -- start start 2024-03-29T11:41:17.777 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.773+0000 7fc8d8a8f640 1 librados: setting wanted keys 2024-03-29T11:41:17.777 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.773+0000 7fc8d8a8f640 1 librados: calling monclient init 2024-03-29T11:41:17.777 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.773+0000 7fc8d8a8f640 10 monclient: init 2024-03-29T11:41:17.778 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.774+0000 7fc8d8a8f640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:17.778 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.774+0000 7fc8d8a8f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:41:17.778 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.774+0000 7fc8d8a8f640 10 monclient(hunting): picked mon.noname-a con 0x7fc8d40ac5a0 addr v1:172.21.15.72:6789/0 2024-03-29T11:41:17.780 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.778+0000 7fc8d8a8f640 10 monclient(hunting): picked mon.noname-b con 0x7fc8d40b0770 addr v1:172.21.15.89:6789/0 2024-03-29T11:41:17.788 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.781+0000 7fc8d1d9b640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc8d40b0770 0x7fc8d41dbaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:48411/0 (socket says 172.21.15.72:48411) 2024-03-29T11:41:17.788 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.781+0000 7fc8d8a8f640 10 monclient(hunting): picked mon.noname-c con 0x7fc8d414a700 addr v1:172.21.15.72:6790/0 2024-03-29T11:41:17.788 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.781+0000 7fc8d1d9b640 1 -- 172.21.15.72:0/265612337 learned_addr learned my addr 172.21.15.72:0/265612337 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:41:17.788 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.781+0000 7fc8d8a8f640 1 -- 172.21.15.72:0/265612337 --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8d4128150 con 0x7fc8d40ac5a0 2024-03-29T11:41:17.788 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.781+0000 7fc8d8a8f640 1 -- 172.21.15.72:0/265612337 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8d41283c0 con 0x7fc8d40b0770 2024-03-29T11:41:17.788 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.781+0000 7fc8d8a8f640 1 -- 172.21.15.72:0/265612337 --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8d4126fa0 con 0x7fc8d414a700 2024-03-29T11:41:17.788 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.781+0000 7fc8d8a8f640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:17.788 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.782+0000 7fc8d8a8f640 10 monclient(hunting): authenticate will time out at 1237.070101s 2024-03-29T11:41:17.788 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.782+0000 7fc8c2ffd640 10 client.?.objecter ms_handle_connect 0x7fc8d40b0770 2024-03-29T11:41:17.788 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.782+0000 7fc8c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:17.788 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.782+0000 7fc8c2ffd640 1 -- 172.21.15.72:0/265612337 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2066052225 0 0) 0x7fc8d41283c0 con 0x7fc8d40b0770 2024-03-29T11:41:17.788 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.782+0000 7fc8c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:17.788 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.782+0000 7fc8c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:17.788 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.782+0000 7fc8c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:17.789 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.782+0000 7fc8c2ffd640 10 monclient(hunting): my global_id is 4417 2024-03-29T11:41:17.789 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.782+0000 7fc8c2ffd640 1 -- 172.21.15.72:0/265612337 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8b4004eb0 con 0x7fc8d40b0770 2024-03-29T11:41:17.789 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.782+0000 7fc8c2ffd640 10 client.?.objecter ms_handle_connect 0x7fc8d40ac5a0 2024-03-29T11:41:17.789 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.782+0000 7fc8c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:17.789 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.783+0000 7fc8c2ffd640 1 -- 172.21.15.72:0/265612337 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1540103571 0 0) 0x7fc8d4128150 con 0x7fc8d40ac5a0 2024-03-29T11:41:17.789 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.783+0000 7fc8c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:17.789 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.783+0000 7fc8c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:17.789 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.783+0000 7fc8c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:17.789 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.783+0000 7fc8c2ffd640 10 monclient(hunting): my global_id is 4449 2024-03-29T11:41:17.789 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.783+0000 7fc8c2ffd640 1 -- 172.21.15.72:0/265612337 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8d41283c0 con 0x7fc8d40ac5a0 2024-03-29T11:41:17.789 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.783+0000 7fc8c2ffd640 1 -- 172.21.15.72:0/265612337 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 861439224 0 0) 0x7fc8b4004eb0 con 0x7fc8d40b0770 2024-03-29T11:41:17.789 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.783+0000 7fc8c2ffd640 1 -- 172.21.15.72:0/265612337 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8d4128150 con 0x7fc8d40b0770 2024-03-29T11:41:17.789 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.783+0000 7fc8c2ffd640 1 -- 172.21.15.72:0/265612337 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fc8c8001900 con 0x7fc8d40b0770 2024-03-29T11:41:17.789 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.783+0000 7fc8c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:17.789 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.783+0000 7fc8c2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:17.789 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.783+0000 7fc8c2ffd640 10 monclient(hunting): dump: 2024-03-29T11:41:17.790 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:41:17.790 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:41:17.790 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:17.790 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:17.790 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:17.790 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:17.790 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:41:17.790 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:41:17.790 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:41:17.790 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:17.790 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.783+0000 7fc8c2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:17.790 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.783+0000 7fc8c2ffd640 1 -- 172.21.15.72:0/265612337 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3927196079 0 0) 0x7fc8d41283c0 con 0x7fc8d40ac5a0 2024-03-29T11:41:17.790 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.783+0000 7fc8c2ffd640 1 -- 172.21.15.72:0/265612337 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8b4004eb0 con 0x7fc8d40ac5a0 2024-03-29T11:41:17.790 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.783+0000 7fc8c2ffd640 1 -- 172.21.15.72:0/265612337 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fc8cc001c50 con 0x7fc8d40ac5a0 2024-03-29T11:41:17.790 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.783+0000 7fc8c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:17.791 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.783+0000 7fc8c2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:17.791 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.783+0000 7fc8c2ffd640 10 monclient(hunting): dump: 2024-03-29T11:41:17.791 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:41:17.791 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:41:17.791 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:17.791 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:17.791 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:17.791 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:17.791 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:41:17.791 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:41:17.791 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:41:17.791 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:17.791 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.783+0000 7fc8c2ffd640 1 -- 172.21.15.72:0/265612337 <== mon.1 v1:172.21.15.89:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2650050385 0 0) 0x7fc8d4128150 con 0x7fc8d40b0770 2024-03-29T11:41:17.791 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.783+0000 7fc8c2ffd640 1 -- 172.21.15.72:0/265612337 >> v1:172.21.15.72:6790/0 conn(0x7fc8d414a700 legacy=0x7fc8d41e7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:17.791 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.783+0000 7fc8c2ffd640 1 -- 172.21.15.72:0/265612337 >> v1:172.21.15.72:6789/0 conn(0x7fc8d40ac5a0 legacy=0x7fc8d41451d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:17.791 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.783+0000 7fc8c2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:41:17.792 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.783+0000 7fc8c2ffd640 1 monclient: found mon.b 2024-03-29T11:41:17.792 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.783+0000 7fc8c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:17.792 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.783+0000 7fc8c2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:41:17.792 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.783+0000 7fc8c2ffd640 1 -- 172.21.15.72:0/265612337 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d4126040 con 0x7fc8d40b0770 2024-03-29T11:41:17.792 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.783+0000 7fc8c2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:41:17.792 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.783+0000 7fc8c2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:41:17.792 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.783+0000 7fc8c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:17.792 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.784+0000 7fc8d8a8f640 5 monclient: authenticate success, global_id 4417 2024-03-29T11:41:17.792 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.784+0000 7fc8d8a8f640 10 monclient: _renew_subs 2024-03-29T11:41:17.792 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.784+0000 7fc8d8a8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:41:17.792 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.784+0000 7fc8d8a8f640 1 -- 172.21.15.72:0/265612337 --> v1:172.21.15.89:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8d41458e0 con 0x7fc8d40b0770 2024-03-29T11:41:17.795 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.784+0000 7fc8c2ffd640 1 -- 172.21.15.72:0/265612337 <== mon.1 v1:172.21.15.89:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8c80047e0 con 0x7fc8d40b0770 2024-03-29T11:41:17.795 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.784+0000 7fc8c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:17.795 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.784+0000 7fc8d8a8f640 10 client.4417.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:17.795 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.784+0000 7fc8d8a8f640 10 monclient: _renew_subs 2024-03-29T11:41:17.795 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.784+0000 7fc8d8a8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:41:17.795 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.784+0000 7fc8d8a8f640 1 -- 172.21.15.72:0/265612337 --> v1:172.21.15.89:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8d4127610 con 0x7fc8d40b0770 2024-03-29T11:41:17.795 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.784+0000 7fc8d8a8f640 1 librados: init done 2024-03-29T11:41:17.795 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.784+0000 7fc8c2ffd640 1 -- 172.21.15.72:0/265612337 <== mon.1 v1:172.21.15.89:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fc8c8004af0 con 0x7fc8d40b0770 2024-03-29T11:41:17.795 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.784+0000 7fc8c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:17.795 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.784+0000 7fc8c2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:41:17.795 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.784+0000 7fc8c2ffd640 10 monclient: dump: 2024-03-29T11:41:17.796 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:41:17.796 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:41:17.796 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:17.796 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:17.796 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:17.796 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:17.796 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:41:17.796 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:41:17.796 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:41:17.796 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:17.796 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.784+0000 7fc8d0d99640 4 set_mon_vals no callback set 2024-03-29T11:41:17.796 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.786+0000 7fc8c2ffd640 1 -- 172.21.15.72:0/265612337 <== mon.1 v1:172.21.15.89:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7fc8c801aac0 con 0x7fc8d40b0770 2024-03-29T11:41:17.796 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.787+0000 7fc8c2ffd640 1 -- 172.21.15.72:0/265612337 <== mon.1 v1:172.21.15.89:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7fc8c8084a90 con 0x7fc8d40b0770 2024-03-29T11:41:17.796 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.787+0000 7fc8c2ffd640 10 client.4417.objecter ms_dispatch 0x7fc8d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:17.796 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.787+0000 7fc8c2ffd640 3 client.4417.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:17.796 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.787+0000 7fc8c2ffd640 3 client.4417.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:17.797 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.787+0000 7fc8c2ffd640 20 client.4417.objecter dump_active .. 0 homeless 2024-03-29T11:41:17.797 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.787+0000 7fc8d259c640 10 monclient: get_auth_request con 0x7fc8b406e050 auth_method 0 2024-03-29T11:41:17.797 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.788+0000 7fc8c2ffd640 10 client.4417.objecter ms_handle_connect 0x7fc8b406e050 2024-03-29T11:41:17.797 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.788+0000 7fc8d8a8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:17.797 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.788+0000 7fc8d8a8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:17.797 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.788+0000 7fc8d8a8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:41:17.797 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.788+0000 7fc8d8a8f640 1 -- 172.21.15.72:0/265612337 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8b8005080 con 0x7fc8d40b0770 2024-03-29T11:41:17.799 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.796+0000 7fc8c2ffd640 1 -- 172.21.15.72:0/265612337 <== mon.1 v1:172.21.15.89:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc8c80501a0 con 0x7fc8d40b0770 2024-03-29T11:41:17.799 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.796+0000 7fc8c2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:17.799 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:17.796+0000 7fc8c2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:18.119 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:18.117+0000 7fc8d8a8f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:18.119 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:18.117+0000 7fc8d8a8f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:18.119 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:18.117+0000 7fc8d8a8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:41:18.119 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:18.117+0000 7fc8d8a8f640 1 -- 172.21.15.72:0/265612337 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc8b8005330 con 0x7fc8d40b0770 2024-03-29T11:41:18.120 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:18.118+0000 7fc8c2ffd640 1 -- 172.21.15.72:0/265612337 <== mon.1 v1:172.21.15.89:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7fc8c80522a0 con 0x7fc8d40b0770 2024-03-29T11:41:18.120 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:18.118+0000 7fc8c2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:18.120 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:18.118+0000 7fc8c2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:18.124 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:41:18.124 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:18.125 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:18.121+0000 7fc8d8a8f640 10 librados: watch_flush enter 2024-03-29T11:41:18.126 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:18.122+0000 7fc8d8a8f640 10 librados: watch_flush exit 2024-03-29T11:41:18.126 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:18.122+0000 7fc8d8a8f640 20 client.4417.objecter shutdown clearing up homeless session... 2024-03-29T11:41:18.126 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:18.122+0000 7fc8d8a8f640 10 client.4417.objecter successfully canceled tick 2024-03-29T11:41:18.126 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:18.122+0000 7fc8d8a8f640 1 -- 172.21.15.72:0/265612337 >> v1:172.21.15.72:6816/1578866102 conn(0x7fc8b406e050 legacy=0x7fc8b4070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:18.126 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:18.122+0000 7fc8d8a8f640 10 monclient: shutdown 2024-03-29T11:41:18.126 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:18.122+0000 7fc8d8a8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:18.126 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:18.122+0000 7fc8d8a8f640 1 -- 172.21.15.72:0/265612337 >> v1:172.21.15.89:6789/0 conn(0x7fc8d40b0770 legacy=0x7fc8d41dbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:18.126 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:18.122+0000 7fc8d8a8f640 1 -- 172.21.15.72:0/265612337 shutdown_connections 2024-03-29T11:41:18.126 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:18.122+0000 7fc8d8a8f640 1 -- 172.21.15.72:0/265612337 >> 172.21.15.72:0/265612337 conn(0x7fc8d406b660 msgr2=0x7fc8d40b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:18.127 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:18.122+0000 7fc8d8a8f640 1 -- 172.21.15.72:0/265612337 shutdown_connections 2024-03-29T11:41:18.127 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:18.122+0000 7fc8d8a8f640 1 -- 172.21.15.72:0/265612337 wait complete. 2024-03-29T11:41:18.127 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:18.122+0000 7fc8d8a8f640 1 librados: shutdown 2024-03-29T11:41:18.143 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:24.950 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:25.088 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.085+0000 7f0b5713b640 10 monclient: get_monmap_and_config 2024-03-29T11:41:25.088 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.085+0000 7f0b5713b640 10 monclient: build_initial_monmap 2024-03-29T11:41:25.088 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.085+0000 7f0b5713b640 10 monclient: monmap: 2024-03-29T11:41:25.088 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:41:25.088 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:41:25.088 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:41:25.086654+0000 2024-03-29T11:41:25.088 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:41:25.086654+0000 2024-03-29T11:41:25.088 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:25.088 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:25.088 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:41:25.088 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:41:25.088 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:41:25.089 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:25.091 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.086+0000 7f0b5713b640 1 Processor -- start 2024-03-29T11:41:25.091 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.086+0000 7f0b5713b640 1 -- start start 2024-03-29T11:41:25.091 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.086+0000 7f0b5713b640 10 monclient: init 2024-03-29T11:41:25.091 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.087+0000 7f0b5713b640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:25.091 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.087+0000 7f0b5713b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:41:25.091 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.087+0000 7f0b5713b640 10 monclient(hunting): picked mon.noname-c con 0x7f0b501457f0 addr v1:172.21.15.89:6789/0 2024-03-29T11:41:25.091 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.087+0000 7f0b5713b640 10 monclient(hunting): picked mon.noname-a con 0x7f0b5013f650 addr v1:172.21.15.72:6789/0 2024-03-29T11:41:25.091 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.087+0000 7f0b5713b640 10 monclient(hunting): picked mon.noname-b con 0x7f0b500a9d70 addr v1:172.21.15.72:6790/0 2024-03-29T11:41:25.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.087+0000 7f0b5713b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b500aaef0 con 0x7f0b5013f650 2024-03-29T11:41:25.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.087+0000 7f0b5713b640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b50145bd0 con 0x7f0b501457f0 2024-03-29T11:41:25.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.087+0000 7f0b5713b640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b50145eb0 con 0x7f0b500a9d70 2024-03-29T11:41:25.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.087+0000 7f0b5713b640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:25.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.087+0000 7f0b5713b640 10 monclient(hunting): authenticate will time out at 1244.375870s 2024-03-29T11:41:25.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.087+0000 7f0b4ffff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f0b5013f650 0x7f0b500b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:46994/0 (socket says 172.21.15.72:46994) 2024-03-29T11:41:25.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.087+0000 7f0b4ffff640 1 -- 172.21.15.72:0/3285226074 learned_addr learned my addr 172.21.15.72:0/3285226074 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:41:25.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.088+0000 7f0b4f7fe640 1 -- 172.21.15.72:0/3285226074 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2351240991 0 0) 0x7f0b500aaef0 con 0x7f0b5013f650 2024-03-29T11:41:25.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.088+0000 7f0b4f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:41:25.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.088+0000 7f0b4f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:25.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.088+0000 7f0b4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:25.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.088+0000 7f0b4f7fe640 10 monclient(hunting): my global_id is 4452 2024-03-29T11:41:25.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.088+0000 7f0b4f7fe640 1 -- 172.21.15.72:0/3285226074 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b38003520 con 0x7f0b5013f650 2024-03-29T11:41:25.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.088+0000 7f0b4f7fe640 1 -- 172.21.15.72:0/3285226074 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 252993183 0 0) 0x7f0b38003520 con 0x7f0b5013f650 2024-03-29T11:41:25.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.088+0000 7f0b4f7fe640 1 -- 172.21.15.72:0/3285226074 >> v1:172.21.15.72:6790/0 conn(0x7f0b500a9d70 legacy=0x7f0b5014d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:25.093 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.091+0000 7f0b4f7fe640 1 -- 172.21.15.72:0/3285226074 >> v1:172.21.15.89:6789/0 conn(0x7f0b501457f0 legacy=0x7f0b5013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:25.094 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.091+0000 7f0b4f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:41:25.094 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.091+0000 7f0b4f7fe640 1 monclient: found mon.noname-a 2024-03-29T11:41:25.094 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.091+0000 7f0b4f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:25.094 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.091+0000 7f0b4f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.72:6789/0 2024-03-29T11:41:25.094 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.091+0000 7f0b4f7fe640 1 -- 172.21.15.72:0/3285226074 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b50146340 con 0x7f0b5013f650 2024-03-29T11:41:25.094 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.091+0000 7f0b4f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:41:25.094 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.091+0000 7f0b4f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:41:25.094 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.091+0000 7f0b4f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:25.094 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.091+0000 7f0b4f7fe640 1 -- 172.21.15.72:0/3285226074 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f0b44001690 con 0x7f0b5013f650 2024-03-29T11:41:25.094 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.091+0000 7f0b4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:25.094 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.091+0000 7f0b4f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:41:25.094 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.091+0000 7f0b4f7fe640 10 monclient: dump: 2024-03-29T11:41:25.094 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:41:25.095 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:41:25.095 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:25.095 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:25.095 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:25.095 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:25.095 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:41:25.095 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:41:25.095 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:41:25.095 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:25.095 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.091+0000 7f0b5713b640 5 monclient: authenticate success, global_id 4452 2024-03-29T11:41:25.095 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.091+0000 7f0b5713b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:25.095 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.091+0000 7f0b4f7fe640 1 -- 172.21.15.72:0/3285226074 <== mon.0 v1:172.21.15.72:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b44001860 con 0x7f0b5013f650 2024-03-29T11:41:25.095 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.091+0000 7f0b4f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:25.095 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.091+0000 7f0b4f7fe640 1 -- 172.21.15.72:0/3285226074 <== mon.0 v1:172.21.15.72:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f0b44001cb0 con 0x7f0b5013f650 2024-03-29T11:41:25.095 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.091+0000 7f0b4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:25.096 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.091+0000 7f0b4f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:25.096 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.091+0000 7f0b4f7fe640 10 monclient: dump: 2024-03-29T11:41:25.096 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:41:25.096 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:41:25.096 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:25.096 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:25.096 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:25.096 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:25.096 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:41:25.096 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:41:25.096 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:41:25.096 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:25.096 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.091+0000 7f0b5713b640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:25.096 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.091+0000 7f0b5713b640 4 set_mon_vals no callback set 2024-03-29T11:41:25.098 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.092+0000 7f0b5713b640 10 monclient: shutdown 2024-03-29T11:41:25.099 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.092+0000 7f0b5713b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:25.099 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.092+0000 7f0b5713b640 1 -- 172.21.15.72:0/3285226074 >> v1:172.21.15.72:6789/0 conn(0x7f0b5013f650 legacy=0x7f0b500b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:25.099 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.092+0000 7f0b5713b640 1 -- 172.21.15.72:0/3285226074 shutdown_connections 2024-03-29T11:41:25.099 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.092+0000 7f0b5713b640 1 -- 172.21.15.72:0/3285226074 >> 172.21.15.72:0/3285226074 conn(0x7f0b50066a00 msgr2=0x7f0b50066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:25.099 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.092+0000 7f0b5713b640 1 -- 172.21.15.72:0/3285226074 shutdown_connections 2024-03-29T11:41:25.099 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.092+0000 7f0b5713b640 1 -- 172.21.15.72:0/3285226074 wait complete. 2024-03-29T11:41:25.099 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.093+0000 7f0b5713b640 10 monclient: build_initial_monmap 2024-03-29T11:41:25.099 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.093+0000 7f0b5713b640 10 monclient: monmap: 2024-03-29T11:41:25.099 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:41:25.099 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:25.099 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:41:25.099 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:41:25.099 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:25.099 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:25.099 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:41:25.100 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:41:25.101 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:41:25.101 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:25.101 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.094+0000 7f0b5713b640 1 librados: starting msgr at 2024-03-29T11:41:25.101 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.094+0000 7f0b5713b640 1 librados: starting objecter 2024-03-29T11:41:25.102 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.094+0000 7f0b5713b640 1 Processor -- start 2024-03-29T11:41:25.102 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.094+0000 7f0b5713b640 1 -- start start 2024-03-29T11:41:25.102 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.094+0000 7f0b5713b640 1 librados: setting wanted keys 2024-03-29T11:41:25.102 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.094+0000 7f0b5713b640 1 librados: calling monclient init 2024-03-29T11:41:25.102 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.094+0000 7f0b5713b640 10 monclient: init 2024-03-29T11:41:25.102 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.096+0000 7f0b5713b640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:25.102 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.096+0000 7f0b5713b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:41:25.102 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.096+0000 7f0b5713b640 10 monclient(hunting): picked mon.noname-c con 0x7f0b500a9d70 addr v1:172.21.15.72:6790/0 2024-03-29T11:41:25.102 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.096+0000 7f0b5713b640 10 monclient(hunting): picked mon.noname-a con 0x7f0b5013f650 addr v1:172.21.15.72:6789/0 2024-03-29T11:41:25.102 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.096+0000 7f0b5713b640 10 monclient(hunting): picked mon.noname-b con 0x7f0b501457f0 addr v1:172.21.15.89:6789/0 2024-03-29T11:41:25.102 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.096+0000 7f0b5713b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b50141910 con 0x7f0b5013f650 2024-03-29T11:41:25.102 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.096+0000 7f0b5713b640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b5013b1b0 con 0x7f0b501457f0 2024-03-29T11:41:25.109 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.096+0000 7f0b5713b640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b50142cb0 con 0x7f0b500a9d70 2024-03-29T11:41:25.109 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.096+0000 7f0b4ffff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f0b5013f650 0x7f0b501425a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50036/0 (socket says 172.21.15.72:50036) 2024-03-29T11:41:25.109 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.096+0000 7f0b4ffff640 1 -- 172.21.15.72:0/3949344454 learned_addr learned my addr 172.21.15.72:0/3949344454 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:41:25.109 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.097+0000 7f0b4d7fa640 10 client.?.objecter ms_handle_connect 0x7f0b5013f650 2024-03-29T11:41:25.109 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.097+0000 7f0b4d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:25.109 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.097+0000 7f0b5713b640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:25.109 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.097+0000 7f0b4d7fa640 10 client.?.objecter ms_handle_connect 0x7f0b501457f0 2024-03-29T11:41:25.109 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.097+0000 7f0b4d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:25.110 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.097+0000 7f0b4d7fa640 1 -- 172.21.15.72:0/3949344454 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3587755135 0 0) 0x7f0b50141910 con 0x7f0b5013f650 2024-03-29T11:41:25.110 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.097+0000 7f0b5713b640 10 monclient(hunting): authenticate will time out at 1244.385454s 2024-03-29T11:41:25.110 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.097+0000 7f0b4d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:41:25.110 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.097+0000 7f0b4d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:25.110 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.097+0000 7f0b4d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:25.110 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.097+0000 7f0b4d7fa640 10 monclient(hunting): my global_id is 4455 2024-03-29T11:41:25.110 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.097+0000 7f0b4d7fa640 1 -- 172.21.15.72:0/3949344454 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b3c004eb0 con 0x7f0b5013f650 2024-03-29T11:41:25.110 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.097+0000 7f0b4d7fa640 1 -- 172.21.15.72:0/3949344454 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 705352040 0 0) 0x7f0b5013b1b0 con 0x7f0b501457f0 2024-03-29T11:41:25.110 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.097+0000 7f0b4d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:41:25.110 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.097+0000 7f0b4d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:25.110 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.097+0000 7f0b4d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:25.110 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.097+0000 7f0b4d7fa640 10 monclient(hunting): my global_id is 4423 2024-03-29T11:41:25.110 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.097+0000 7f0b4d7fa640 1 -- 172.21.15.72:0/3949344454 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b50141910 con 0x7f0b501457f0 2024-03-29T11:41:25.110 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.098+0000 7f0b4d7fa640 1 -- 172.21.15.72:0/3949344454 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1992487128 0 0) 0x7f0b50141910 con 0x7f0b501457f0 2024-03-29T11:41:25.111 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.098+0000 7f0b4d7fa640 1 -- 172.21.15.72:0/3949344454 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b5013b1b0 con 0x7f0b501457f0 2024-03-29T11:41:25.111 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.098+0000 7f0b4d7fa640 1 -- 172.21.15.72:0/3949344454 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f0b48001b30 con 0x7f0b501457f0 2024-03-29T11:41:25.111 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.098+0000 7f0b4d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:25.111 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.098+0000 7f0b4d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:25.111 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.098+0000 7f0b4d7fa640 10 monclient(hunting): dump: 2024-03-29T11:41:25.111 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:41:25.111 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:41:25.111 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:25.111 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:25.111 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:25.111 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:25.111 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:41:25.111 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:41:25.111 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:41:25.111 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:25.112 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.098+0000 7f0b4d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:25.112 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.098+0000 7f0b4ffff640 1 -- 172.21.15.72:0/3949344454 >> v1:172.21.15.72:6789/0 conn(0x7f0b5013f650 legacy=0x7f0b501425a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-29T11:41:25.112 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.098+0000 7f0b4ffff640 1 -- 172.21.15.72:0/3949344454 >> v1:172.21.15.72:6789/0 conn(0x7f0b5013f650 legacy=0x7f0b501425a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:41:25.112 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.098+0000 7f0b4ffff640 1 --1- 172.21.15.72:0/3949344454 >> v1:172.21.15.72:6789/0 conn(0x7f0b5013f650 0x7f0b501425a0 :-1 s=OPENED pgs=613258 cs=1 l=1).handle_message read tag failed 2024-03-29T11:41:25.112 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.098+0000 7f0b4ffff640 1 --1- 172.21.15.72:0/3949344454 >> v1:172.21.15.72:6789/0 conn(0x7f0b5013f650 0x7f0b501425a0 :-1 s=OPENED pgs=613258 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:41:25.112 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.098+0000 7f0b4d7fa640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.72:6789/0 2024-03-29T11:41:25.112 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.099+0000 7f0b4d7fa640 1 -- 172.21.15.72:0/3949344454 <== mon.1 v1:172.21.15.89:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3063806610 0 0) 0x7f0b5013b1b0 con 0x7f0b501457f0 2024-03-29T11:41:25.112 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.099+0000 7f0b4d7fa640 1 -- 172.21.15.72:0/3949344454 >> v1:172.21.15.72:6790/0 conn(0x7f0b500a9d70 legacy=0x7f0b50140330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:41:25.112 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.100+0000 7f0b4d7fa640 1 -- 172.21.15.72:0/3949344454 >> v1:172.21.15.72:6789/0 conn(0x7f0b5013f650 legacy=0x7f0b501425a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:41:25.112 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.100+0000 7f0b4d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:41:25.112 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.100+0000 7f0b4d7fa640 1 monclient: found mon.b 2024-03-29T11:41:25.112 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.100+0000 7f0b4d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:25.112 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.100+0000 7f0b4d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:41:25.112 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.100+0000 7f0b4d7fa640 1 -- 172.21.15.72:0/3949344454 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b5013fda0 con 0x7f0b501457f0 2024-03-29T11:41:25.112 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.100+0000 7f0b4d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:41:25.113 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.100+0000 7f0b4d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:41:25.113 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.100+0000 7f0b4d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:25.113 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.100+0000 7f0b5713b640 5 monclient: authenticate success, global_id 4423 2024-03-29T11:41:25.113 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.100+0000 7f0b5713b640 10 monclient: _renew_subs 2024-03-29T11:41:25.113 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.100+0000 7f0b5713b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:41:25.113 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.100+0000 7f0b5713b640 1 -- 172.21.15.72:0/3949344454 --> v1:172.21.15.89:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b501e3050 con 0x7f0b501457f0 2024-03-29T11:41:25.113 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.100+0000 7f0b5713b640 10 client.4423.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:25.113 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.100+0000 7f0b5713b640 10 monclient: _renew_subs 2024-03-29T11:41:25.113 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.101+0000 7f0b5713b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:41:25.113 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.101+0000 7f0b5713b640 1 -- 172.21.15.72:0/3949344454 --> v1:172.21.15.89:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b501e33c0 con 0x7f0b501457f0 2024-03-29T11:41:25.113 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.101+0000 7f0b4d7fa640 1 -- 172.21.15.72:0/3949344454 <== mon.1 v1:172.21.15.89:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b48002040 con 0x7f0b501457f0 2024-03-29T11:41:25.113 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.101+0000 7f0b4d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:25.113 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.101+0000 7f0b4d7fa640 1 -- 172.21.15.72:0/3949344454 <== mon.1 v1:172.21.15.89:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f0b48003b10 con 0x7f0b501457f0 2024-03-29T11:41:25.113 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.101+0000 7f0b4d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:25.113 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.101+0000 7f0b4d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:41:25.114 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.101+0000 7f0b4d7fa640 10 monclient: dump: 2024-03-29T11:41:25.114 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:41:25.114 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:41:25.114 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:25.114 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:25.114 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:25.114 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:25.114 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:41:25.114 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:41:25.114 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:41:25.114 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:25.114 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.101+0000 7f0b5713b640 1 librados: init done 2024-03-29T11:41:25.114 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.102+0000 7f0b5713b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:25.114 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.102+0000 7f0b5713b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:25.114 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.102+0000 7f0b5713b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:41:25.115 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.102+0000 7f0b5713b640 1 -- 172.21.15.72:0/3949344454 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b501423f0 con 0x7f0b501457f0 2024-03-29T11:41:25.115 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.102+0000 7f0b4effd640 4 set_mon_vals no callback set 2024-03-29T11:41:25.115 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.103+0000 7f0b4d7fa640 1 -- 172.21.15.72:0/3949344454 <== mon.1 v1:172.21.15.89:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f0b48019ae0 con 0x7f0b501457f0 2024-03-29T11:41:25.115 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.104+0000 7f0b4d7fa640 1 -- 172.21.15.72:0/3949344454 <== mon.1 v1:172.21.15.89:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f0b480839f0 con 0x7f0b501457f0 2024-03-29T11:41:25.115 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.104+0000 7f0b4d7fa640 10 client.4423.objecter ms_dispatch 0x7f0b50000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:25.115 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.104+0000 7f0b4d7fa640 3 client.4423.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:25.115 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.104+0000 7f0b4d7fa640 3 client.4423.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:25.115 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.104+0000 7f0b4d7fa640 20 client.4423.objecter dump_active .. 0 homeless 2024-03-29T11:41:25.115 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.104+0000 7f0b54ed7640 10 monclient: get_auth_request con 0x7f0b3c06dcb0 auth_method 0 2024-03-29T11:41:25.115 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.104+0000 7f0b4d7fa640 10 client.4423.objecter ms_handle_connect 0x7f0b3c06dcb0 2024-03-29T11:41:25.116 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.112+0000 7f0b4d7fa640 1 -- 172.21.15.72:0/3949344454 <== mon.1 v1:172.21.15.89:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0b4804f2d0 con 0x7f0b501457f0 2024-03-29T11:41:25.116 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.112+0000 7f0b4d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:25.116 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.112+0000 7f0b4d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:25.432 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.429+0000 7f0b5713b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:25.432 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.429+0000 7f0b5713b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:25.432 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.429+0000 7f0b5713b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:41:25.432 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.429+0000 7f0b5713b640 1 -- 172.21.15.72:0/3949344454 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b50075f10 con 0x7f0b501457f0 2024-03-29T11:41:25.434 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.432+0000 7f0b4d7fa640 1 -- 172.21.15.72:0/3949344454 <== mon.1 v1:172.21.15.89:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f0b48051200 con 0x7f0b501457f0 2024-03-29T11:41:25.434 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.432+0000 7f0b4d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:25.435 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.432+0000 7f0b4d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:25.435 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:41:25.435 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:25.438 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.435+0000 7f0b5713b640 10 librados: watch_flush enter 2024-03-29T11:41:25.438 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.436+0000 7f0b5713b640 10 librados: watch_flush exit 2024-03-29T11:41:25.438 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.436+0000 7f0b5713b640 20 client.4423.objecter shutdown clearing up homeless session... 2024-03-29T11:41:25.438 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.436+0000 7f0b5713b640 10 client.4423.objecter successfully canceled tick 2024-03-29T11:41:25.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.436+0000 7f0b5713b640 1 -- 172.21.15.72:0/3949344454 >> v1:172.21.15.72:6816/1578866102 conn(0x7f0b3c06dcb0 legacy=0x7f0b3c070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:25.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.436+0000 7f0b5713b640 10 monclient: shutdown 2024-03-29T11:41:25.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.436+0000 7f0b5713b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:25.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.436+0000 7f0b5713b640 1 -- 172.21.15.72:0/3949344454 >> v1:172.21.15.89:6789/0 conn(0x7f0b501457f0 legacy=0x7f0b501e1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:25.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.436+0000 7f0b5713b640 1 -- 172.21.15.72:0/3949344454 shutdown_connections 2024-03-29T11:41:25.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.436+0000 7f0b5713b640 1 -- 172.21.15.72:0/3949344454 >> 172.21.15.72:0/3949344454 conn(0x7f0b50066a00 msgr2=0x7f0b500b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:25.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.437+0000 7f0b5713b640 1 -- 172.21.15.72:0/3949344454 shutdown_connections 2024-03-29T11:41:25.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.437+0000 7f0b5713b640 1 -- 172.21.15.72:0/3949344454 wait complete. 2024-03-29T11:41:25.440 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:25.438+0000 7f0b5713b640 1 librados: shutdown 2024-03-29T11:41:25.457 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:32.265 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:32.419 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.417+0000 7f6fdfa3e640 10 monclient: get_monmap_and_config 2024-03-29T11:41:32.420 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.417+0000 7f6fdfa3e640 10 monclient: build_initial_monmap 2024-03-29T11:41:32.420 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.417+0000 7f6fdfa3e640 10 monclient: monmap: 2024-03-29T11:41:32.420 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:41:32.420 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:41:32.420 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:41:32.418459+0000 2024-03-29T11:41:32.420 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:41:32.418459+0000 2024-03-29T11:41:32.420 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:32.420 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:32.420 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:41:32.420 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:41:32.420 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:41:32.420 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:32.421 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.418+0000 7f6fdfa3e640 1 Processor -- start 2024-03-29T11:41:32.421 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.418+0000 7f6fdfa3e640 1 -- start start 2024-03-29T11:41:32.421 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.418+0000 7f6fdfa3e640 10 monclient: init 2024-03-29T11:41:32.421 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.419+0000 7f6fdfa3e640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:32.421 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.419+0000 7f6fdfa3e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:41:32.422 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.419+0000 7f6fdfa3e640 10 monclient(hunting): picked mon.noname-c con 0x7f6fd0006810 addr v1:172.21.15.89:6789/0 2024-03-29T11:41:32.422 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.419+0000 7f6fdfa3e640 10 monclient(hunting): picked mon.noname-b con 0x7f6fd0006bf0 addr v1:172.21.15.72:6790/0 2024-03-29T11:41:32.422 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.419+0000 7f6fdfa3e640 10 monclient(hunting): picked mon.noname-a con 0x7f6fd00c78f0 addr v1:172.21.15.72:6789/0 2024-03-29T11:41:32.422 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.419+0000 7f6fdfa3e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd00c7f30 con 0x7f6fd00c78f0 2024-03-29T11:41:32.422 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.419+0000 7f6fdfa3e640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd0036fd0 con 0x7f6fd0006810 2024-03-29T11:41:32.422 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.419+0000 7f6fdfa3e640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd00c54f0 con 0x7f6fd0006bf0 2024-03-29T11:41:32.423 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.419+0000 7f6fdfa3e640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:32.423 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.419+0000 7f6fdfa3e640 10 monclient(hunting): authenticate will time out at 1251.707639s 2024-03-29T11:41:32.423 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.419+0000 7f6fdcfd9640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f6fd0006bf0 0x7f6fd00cae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:50026/0 (socket says 172.21.15.72:50026) 2024-03-29T11:41:32.423 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.419+0000 7f6fdcfd9640 1 -- 172.21.15.72:0/54127410 learned_addr learned my addr 172.21.15.72:0/54127410 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:41:32.423 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.420+0000 7f6fcbfff640 1 -- 172.21.15.72:0/54127410 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3113923042 0 0) 0x7f6fd00c54f0 con 0x7f6fd0006bf0 2024-03-29T11:41:32.423 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.420+0000 7f6fcbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:41:32.423 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.420+0000 7f6fcbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:32.423 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.420+0000 7f6fcbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:32.423 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.420+0000 7f6fcbfff640 10 monclient(hunting): my global_id is 4472 2024-03-29T11:41:32.423 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.420+0000 7f6fcbfff640 1 -- 172.21.15.72:0/54127410 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fc4003520 con 0x7f6fd0006bf0 2024-03-29T11:41:32.427 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.420+0000 7f6fcbfff640 1 -- 172.21.15.72:0/54127410 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 117585681 0 0) 0x7f6fd00c7f30 con 0x7f6fd00c78f0 2024-03-29T11:41:32.427 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.420+0000 7f6fcbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:41:32.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.420+0000 7f6fcbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:32.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.420+0000 7f6fcbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:32.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.420+0000 7f6fcbfff640 10 monclient(hunting): my global_id is 4458 2024-03-29T11:41:32.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.420+0000 7f6fcbfff640 1 -- 172.21.15.72:0/54127410 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fd00c54f0 con 0x7f6fd00c78f0 2024-03-29T11:41:32.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.421+0000 7f6fcbfff640 1 -- 172.21.15.72:0/54127410 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2912527666 0 0) 0x7f6fd0036fd0 con 0x7f6fd0006810 2024-03-29T11:41:32.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.421+0000 7f6fcbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:41:32.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.421+0000 7f6fcbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:32.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.421+0000 7f6fcbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:32.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.421+0000 7f6fcbfff640 10 monclient(hunting): my global_id is 4426 2024-03-29T11:41:32.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.421+0000 7f6fcbfff640 1 -- 172.21.15.72:0/54127410 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fd00c7f30 con 0x7f6fd0006810 2024-03-29T11:41:32.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.421+0000 7f6fcbfff640 1 -- 172.21.15.72:0/54127410 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2099978773 0 0) 0x7f6fc4003520 con 0x7f6fd0006bf0 2024-03-29T11:41:32.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.421+0000 7f6fcbfff640 1 -- 172.21.15.72:0/54127410 >> v1:172.21.15.89:6789/0 conn(0x7f6fd0006810 legacy=0x7f6fd0033620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:32.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.421+0000 7f6fcbfff640 1 -- 172.21.15.72:0/54127410 >> v1:172.21.15.72:6789/0 conn(0x7f6fd00c78f0 legacy=0x7f6fd00d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:32.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.421+0000 7f6fcbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:41:32.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.421+0000 7f6fcbfff640 1 monclient: found mon.noname-b 2024-03-29T11:41:32.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.421+0000 7f6fcbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:32.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.421+0000 7f6fcbfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.72:6790/0 2024-03-29T11:41:32.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.421+0000 7f6fcbfff640 1 -- 172.21.15.72:0/54127410 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd0037240 con 0x7f6fd0006bf0 2024-03-29T11:41:32.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.421+0000 7f6fcbfff640 10 monclient: _finish_auth 0 2024-03-29T11:41:32.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.421+0000 7f6fcbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:41:32.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.421+0000 7f6fcbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:32.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.421+0000 7f6fdfa3e640 5 monclient: authenticate success, global_id 4472 2024-03-29T11:41:32.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.421+0000 7f6fcbfff640 1 -- 172.21.15.72:0/54127410 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f6fcc001690 con 0x7f6fd0006bf0 2024-03-29T11:41:32.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.421+0000 7f6fdfa3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:32.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.421+0000 7f6fcbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:32.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.421+0000 7f6fcbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:32.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.421+0000 7f6fcbfff640 10 monclient: dump: 2024-03-29T11:41:32.429 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:41:32.429 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:41:32.429 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:32.429 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:32.430 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:32.430 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:32.430 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:41:32.430 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:41:32.430 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:41:32.430 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:32.430 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.421+0000 7f6fcbfff640 1 -- 172.21.15.72:0/54127410 <== mon.2 v1:172.21.15.72:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6fcc001870 con 0x7f6fd0006bf0 2024-03-29T11:41:32.430 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.421+0000 7f6fcbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:32.430 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.421+0000 7f6fcbfff640 1 -- 172.21.15.72:0/54127410 <== mon.2 v1:172.21.15.72:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f6fcc001c00 con 0x7f6fd0006bf0 2024-03-29T11:41:32.430 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.421+0000 7f6fdfa3e640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:32.430 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.421+0000 7f6fdfa3e640 4 set_mon_vals no callback set 2024-03-29T11:41:32.430 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.421+0000 7f6fcbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:32.430 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.421+0000 7f6fcbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:32.430 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.421+0000 7f6fcbfff640 10 monclient: dump: 2024-03-29T11:41:32.431 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:41:32.431 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:41:32.431 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:32.431 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:32.431 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:32.431 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:32.431 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:41:32.431 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:41:32.431 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:41:32.431 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:32.431 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.422+0000 7f6fdfa3e640 10 monclient: shutdown 2024-03-29T11:41:32.431 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.422+0000 7f6fdfa3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:32.431 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.422+0000 7f6fdfa3e640 1 -- 172.21.15.72:0/54127410 >> v1:172.21.15.72:6790/0 conn(0x7f6fd0006bf0 legacy=0x7f6fd00cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:32.431 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.422+0000 7f6fdfa3e640 1 -- 172.21.15.72:0/54127410 shutdown_connections 2024-03-29T11:41:32.431 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.422+0000 7f6fdfa3e640 1 -- 172.21.15.72:0/54127410 >> 172.21.15.72:0/54127410 conn(0x7f6fd0005c40 msgr2=0x7f6fd0006050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:32.431 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.423+0000 7f6fdfa3e640 1 -- 172.21.15.72:0/54127410 shutdown_connections 2024-03-29T11:41:32.432 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.423+0000 7f6fdfa3e640 1 -- 172.21.15.72:0/54127410 wait complete. 2024-03-29T11:41:32.432 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.423+0000 7f6fdfa3e640 10 monclient: build_initial_monmap 2024-03-29T11:41:32.432 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.423+0000 7f6fdfa3e640 10 monclient: monmap: 2024-03-29T11:41:32.432 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:41:32.432 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:32.432 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:41:32.432 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:41:32.432 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:32.432 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:32.432 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:41:32.432 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:41:32.432 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:41:32.432 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:32.432 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.424+0000 7f6fdfa3e640 1 librados: starting msgr at 2024-03-29T11:41:32.432 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.424+0000 7f6fdfa3e640 1 librados: starting objecter 2024-03-29T11:41:32.432 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.424+0000 7f6fdfa3e640 1 Processor -- start 2024-03-29T11:41:32.433 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.424+0000 7f6fdfa3e640 1 -- start start 2024-03-29T11:41:32.433 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.424+0000 7f6fdfa3e640 1 librados: setting wanted keys 2024-03-29T11:41:32.433 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.424+0000 7f6fdfa3e640 1 librados: calling monclient init 2024-03-29T11:41:32.433 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.424+0000 7f6fdfa3e640 10 monclient: init 2024-03-29T11:41:32.433 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.425+0000 7f6fdfa3e640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:32.433 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.425+0000 7f6fdfa3e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:41:32.438 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.425+0000 7f6fdfa3e640 10 monclient(hunting): picked mon.noname-b con 0x7f6fd0006810 addr v1:172.21.15.89:6789/0 2024-03-29T11:41:32.438 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.425+0000 7f6fdfa3e640 10 monclient(hunting): picked mon.noname-a con 0x7f6fd00c78f0 addr v1:172.21.15.72:6789/0 2024-03-29T11:41:32.438 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.425+0000 7f6fdfa3e640 10 monclient(hunting): picked mon.noname-c con 0x7f6fd002fee0 addr v1:172.21.15.72:6790/0 2024-03-29T11:41:32.438 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.425+0000 7f6fdfa3e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd015a4b0 con 0x7f6fd00c78f0 2024-03-29T11:41:32.438 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.425+0000 7f6fdfa3e640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd00c3500 con 0x7f6fd0006810 2024-03-29T11:41:32.438 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.425+0000 7f6fdfa3e640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd00c3690 con 0x7f6fd002fee0 2024-03-29T11:41:32.438 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.425+0000 7f6fdfa3e640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:32.438 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.425+0000 7f6fdfa3e640 10 monclient(hunting): authenticate will time out at 1251.713960s 2024-03-29T11:41:32.438 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.426+0000 7f6fdd7da640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f6fd0006810 0x7f6fd015fef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:52096/0 (socket says 172.21.15.72:52096) 2024-03-29T11:41:32.438 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.426+0000 7f6fdd7da640 1 -- 172.21.15.72:0/1930739598 learned_addr learned my addr 172.21.15.72:0/1930739598 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:41:32.438 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.426+0000 7f6fc9ffb640 10 client.?.objecter ms_handle_connect 0x7f6fd0006810 2024-03-29T11:41:32.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.426+0000 7f6fc9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:32.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.426+0000 7f6fc9ffb640 10 client.?.objecter ms_handle_connect 0x7f6fd002fee0 2024-03-29T11:41:32.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.426+0000 7f6fc9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:32.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.427+0000 7f6fc9ffb640 10 client.?.objecter ms_handle_connect 0x7f6fd00c78f0 2024-03-29T11:41:32.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.427+0000 7f6fc9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:32.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.427+0000 7f6fc9ffb640 1 -- 172.21.15.72:0/1930739598 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2667217994 0 0) 0x7f6fd00c3500 con 0x7f6fd0006810 2024-03-29T11:41:32.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.427+0000 7f6fc9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:41:32.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.427+0000 7f6fc9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:32.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.427+0000 7f6fc9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:32.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.427+0000 7f6fc9ffb640 10 monclient(hunting): my global_id is 4429 2024-03-29T11:41:32.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.427+0000 7f6fc9ffb640 1 -- 172.21.15.72:0/1930739598 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fbc004eb0 con 0x7f6fd0006810 2024-03-29T11:41:32.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.427+0000 7f6fc9ffb640 1 -- 172.21.15.72:0/1930739598 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1067465057 0 0) 0x7f6fd00c3690 con 0x7f6fd002fee0 2024-03-29T11:41:32.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.427+0000 7f6fc9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:41:32.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.427+0000 7f6fc9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:32.440 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.427+0000 7f6fc9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:32.440 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.427+0000 7f6fc9ffb640 10 monclient(hunting): my global_id is 4475 2024-03-29T11:41:32.440 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.427+0000 7f6fc9ffb640 1 -- 172.21.15.72:0/1930739598 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fd00c3500 con 0x7f6fd002fee0 2024-03-29T11:41:32.440 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.427+0000 7f6fc9ffb640 1 -- 172.21.15.72:0/1930739598 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 663487263 0 0) 0x7f6fbc004eb0 con 0x7f6fd0006810 2024-03-29T11:41:32.440 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.427+0000 7f6fc9ffb640 1 -- 172.21.15.72:0/1930739598 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6fd00c3690 con 0x7f6fd0006810 2024-03-29T11:41:32.440 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.427+0000 7f6fc9ffb640 1 -- 172.21.15.72:0/1930739598 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f6fd40019c0 con 0x7f6fd0006810 2024-03-29T11:41:32.440 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.427+0000 7f6fc9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:32.440 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.427+0000 7f6fc9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:32.440 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.427+0000 7f6fc9ffb640 10 monclient(hunting): dump: 2024-03-29T11:41:32.440 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:41:32.440 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:41:32.440 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:32.440 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:32.440 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:32.440 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:32.441 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:41:32.441 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:41:32.441 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:41:32.441 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:32.441 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.427+0000 7f6fc9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:32.441 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.428+0000 7f6fc9ffb640 1 -- 172.21.15.72:0/1930739598 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 598689182 0 0) 0x7f6fd015a4b0 con 0x7f6fd00c78f0 2024-03-29T11:41:32.441 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.428+0000 7f6fc9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:41:32.441 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.428+0000 7f6fc9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:32.441 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.428+0000 7f6fc9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:32.441 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.428+0000 7f6fc9ffb640 10 monclient(hunting): my global_id is 4461 2024-03-29T11:41:32.441 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.428+0000 7f6fc9ffb640 1 -- 172.21.15.72:0/1930739598 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fbc004eb0 con 0x7f6fd00c78f0 2024-03-29T11:41:32.441 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.428+0000 7f6fc9ffb640 1 -- 172.21.15.72:0/1930739598 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4015432902 0 0) 0x7f6fd00c3500 con 0x7f6fd002fee0 2024-03-29T11:41:32.441 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.428+0000 7f6fc9ffb640 1 -- 172.21.15.72:0/1930739598 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6fd015a4b0 con 0x7f6fd002fee0 2024-03-29T11:41:32.441 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.428+0000 7f6fc9ffb640 1 -- 172.21.15.72:0/1930739598 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f6fd804fdd0 con 0x7f6fd002fee0 2024-03-29T11:41:32.441 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.428+0000 7f6fc9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:32.441 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.428+0000 7f6fc9ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:32.442 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.428+0000 7f6fc9ffb640 10 monclient(hunting): dump: 2024-03-29T11:41:32.442 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:41:32.442 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:41:32.442 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:32.442 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:32.442 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:32.442 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:32.442 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:41:32.442 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:41:32.442 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:41:32.442 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:32.442 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.428+0000 7f6fc9ffb640 1 -- 172.21.15.72:0/1930739598 <== mon.1 v1:172.21.15.89:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3321204640 0 0) 0x7f6fd00c3690 con 0x7f6fd0006810 2024-03-29T11:41:32.442 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.428+0000 7f6fc9ffb640 1 -- 172.21.15.72:0/1930739598 >> v1:172.21.15.72:6790/0 conn(0x7f6fd002fee0 legacy=0x7f6fd0168290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:32.442 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.428+0000 7f6fc9ffb640 1 -- 172.21.15.72:0/1930739598 >> v1:172.21.15.72:6789/0 conn(0x7f6fd00c78f0 legacy=0x7f6fd00bf550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:32.442 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.428+0000 7f6fc9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:41:32.443 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.428+0000 7f6fc9ffb640 1 monclient: found mon.b 2024-03-29T11:41:32.443 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.428+0000 7f6fc9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:32.443 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.428+0000 7f6fc9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:41:32.443 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.428+0000 7f6fc9ffb640 1 -- 172.21.15.72:0/1930739598 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd0160600 con 0x7f6fd0006810 2024-03-29T11:41:32.443 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.428+0000 7f6fc9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:41:32.443 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.428+0000 7f6fc9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:41:32.443 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.428+0000 7f6fc9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:32.443 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.428+0000 7f6fdfa3e640 5 monclient: authenticate success, global_id 4429 2024-03-29T11:41:32.443 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.428+0000 7f6fdfa3e640 10 monclient: _renew_subs 2024-03-29T11:41:32.443 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.428+0000 7f6fdfa3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:41:32.443 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.428+0000 7f6fdfa3e640 1 -- 172.21.15.72:0/1930739598 --> v1:172.21.15.89:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fd01607f0 con 0x7f6fd0006810 2024-03-29T11:41:32.443 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.428+0000 7f6fdfa3e640 10 client.4429.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:32.443 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.428+0000 7f6fdfa3e640 10 monclient: _renew_subs 2024-03-29T11:41:32.443 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.428+0000 7f6fdfa3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:41:32.443 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.428+0000 7f6fdfa3e640 1 -- 172.21.15.72:0/1930739598 --> v1:172.21.15.89:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fd00bfe40 con 0x7f6fd0006810 2024-03-29T11:41:32.444 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.428+0000 7f6fdfa3e640 1 librados: init done 2024-03-29T11:41:32.444 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.429+0000 7f6fc9ffb640 1 -- 172.21.15.72:0/1930739598 <== mon.1 v1:172.21.15.89:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6fd4001c80 con 0x7f6fd0006810 2024-03-29T11:41:32.444 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.429+0000 7f6fc9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:32.444 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.429+0000 7f6fcb7fe640 4 set_mon_vals no callback set 2024-03-29T11:41:32.444 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.431+0000 7f6fc9ffb640 1 -- 172.21.15.72:0/1930739598 <== mon.1 v1:172.21.15.89:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f6fd4003a60 con 0x7f6fd0006810 2024-03-29T11:41:32.444 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.431+0000 7f6fc9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:32.444 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.431+0000 7f6fc9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:41:32.444 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.431+0000 7f6fc9ffb640 10 monclient: dump: 2024-03-29T11:41:32.444 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:41:32.444 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:41:32.444 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:32.444 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:32.444 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:32.444 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:32.444 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:41:32.445 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:41:32.445 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:41:32.445 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:32.445 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.432+0000 7f6fc9ffb640 1 -- 172.21.15.72:0/1930739598 <== mon.1 v1:172.21.15.89:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f6fd4003c80 con 0x7f6fd0006810 2024-03-29T11:41:32.445 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.432+0000 7f6fdfa3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:32.445 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.432+0000 7f6fdcfd9640 10 monclient: get_auth_request con 0x7f6fbc06ddb0 auth_method 0 2024-03-29T11:41:32.445 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.432+0000 7f6fc9ffb640 1 -- 172.21.15.72:0/1930739598 <== mon.1 v1:172.21.15.89:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f6fd4084b60 con 0x7f6fd0006810 2024-03-29T11:41:32.445 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.432+0000 7f6fc9ffb640 10 client.4429.objecter ms_dispatch 0x7f6fd8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:32.445 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.432+0000 7f6fc9ffb640 3 client.4429.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:32.445 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.432+0000 7f6fc9ffb640 3 client.4429.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:32.445 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.432+0000 7f6fc9ffb640 20 client.4429.objecter dump_active .. 0 homeless 2024-03-29T11:41:32.445 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.433+0000 7f6fdfa3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:32.445 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.433+0000 7f6fdfa3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:41:32.445 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.433+0000 7f6fdfa3e640 1 -- 172.21.15.72:0/1930739598 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fb8005080 con 0x7f6fd0006810 2024-03-29T11:41:32.445 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.433+0000 7f6fc9ffb640 10 client.4429.objecter ms_handle_connect 0x7f6fbc06ddb0 2024-03-29T11:41:32.446 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.440+0000 7f6fc9ffb640 1 -- 172.21.15.72:0/1930739598 <== mon.1 v1:172.21.15.89:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6fd40504c0 con 0x7f6fd0006810 2024-03-29T11:41:32.446 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.441+0000 7f6fc9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:32.446 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.441+0000 7f6fc9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:32.830 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.827+0000 7f6fdfa3e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:32.830 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.827+0000 7f6fdfa3e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:32.830 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.827+0000 7f6fdfa3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:41:32.830 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.827+0000 7f6fdfa3e640 1 -- 172.21.15.72:0/1930739598 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6fb8005330 con 0x7f6fd0006810 2024-03-29T11:41:32.831 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.828+0000 7f6fc9ffb640 1 -- 172.21.15.72:0/1930739598 <== mon.1 v1:172.21.15.89:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f6fd4052370 con 0x7f6fd0006810 2024-03-29T11:41:32.831 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.828+0000 7f6fc9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:32.831 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.828+0000 7f6fc9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:32.831 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:41:32.831 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:32.834 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.832+0000 7f6fdfa3e640 10 librados: watch_flush enter 2024-03-29T11:41:32.834 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.832+0000 7f6fdfa3e640 10 librados: watch_flush exit 2024-03-29T11:41:32.834 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.832+0000 7f6fdfa3e640 20 client.4429.objecter shutdown clearing up homeless session... 2024-03-29T11:41:32.835 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.832+0000 7f6fdfa3e640 10 client.4429.objecter successfully canceled tick 2024-03-29T11:41:32.835 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.832+0000 7f6fdfa3e640 1 -- 172.21.15.72:0/1930739598 >> v1:172.21.15.72:6816/1578866102 conn(0x7f6fbc06ddb0 legacy=0x7f6fbc070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:32.835 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.832+0000 7f6fdfa3e640 10 monclient: shutdown 2024-03-29T11:41:32.835 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.832+0000 7f6fdfa3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:32.835 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.832+0000 7f6fdfa3e640 1 -- 172.21.15.72:0/1930739598 >> v1:172.21.15.89:6789/0 conn(0x7f6fd0006810 legacy=0x7f6fd015fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:32.836 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.832+0000 7f6fdfa3e640 1 -- 172.21.15.72:0/1930739598 shutdown_connections 2024-03-29T11:41:32.836 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.832+0000 7f6fdfa3e640 1 -- 172.21.15.72:0/1930739598 >> 172.21.15.72:0/1930739598 conn(0x7f6fd0005c40 msgr2=0x7f6fd0034b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:32.836 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.833+0000 7f6fdfa3e640 1 -- 172.21.15.72:0/1930739598 shutdown_connections 2024-03-29T11:41:32.836 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.833+0000 7f6fdfa3e640 1 -- 172.21.15.72:0/1930739598 wait complete. 2024-03-29T11:41:32.836 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:32.833+0000 7f6fdfa3e640 1 librados: shutdown 2024-03-29T11:41:32.852 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:39.659 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:39.812 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.809+0000 7fa504fdc640 10 monclient: get_monmap_and_config 2024-03-29T11:41:39.812 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.809+0000 7fa504fdc640 10 monclient: build_initial_monmap 2024-03-29T11:41:39.812 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.809+0000 7fa504fdc640 10 monclient: monmap: 2024-03-29T11:41:39.812 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:41:39.813 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:41:39.813 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:41:39.810920+0000 2024-03-29T11:41:39.813 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:41:39.810920+0000 2024-03-29T11:41:39.813 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:39.813 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:39.813 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:41:39.813 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:41:39.813 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:41:39.813 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:39.815 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.811+0000 7fa504fdc640 1 Processor -- start 2024-03-29T11:41:39.815 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.811+0000 7fa504fdc640 1 -- start start 2024-03-29T11:41:39.816 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.811+0000 7fa504fdc640 10 monclient: init 2024-03-29T11:41:39.816 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.812+0000 7fa504fdc640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:39.816 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.812+0000 7fa504fdc640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:41:39.816 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.812+0000 7fa504fdc640 10 monclient(hunting): picked mon.noname-a con 0x7fa4f8006d80 addr v1:172.21.15.72:6789/0 2024-03-29T11:41:39.816 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.812+0000 7fa504fdc640 10 monclient(hunting): picked mon.noname-c con 0x7fa4f80cc310 addr v1:172.21.15.89:6789/0 2024-03-29T11:41:39.816 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.812+0000 7fa504fdc640 10 monclient(hunting): picked mon.noname-b con 0x7fa4f80c9430 addr v1:172.21.15.72:6790/0 2024-03-29T11:41:39.816 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.812+0000 7fa504fdc640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4f8008470 con 0x7fa4f8006d80 2024-03-29T11:41:39.816 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.812+0000 7fa504fdc640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4f802be30 con 0x7fa4f80cc310 2024-03-29T11:41:39.816 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.812+0000 7fa504fdc640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4f802c110 con 0x7fa4f80c9430 2024-03-29T11:41:39.816 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.812+0000 7fa504fdc640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:39.816 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.812+0000 7fa504fdc640 10 monclient(hunting): authenticate will time out at 1259.100144s 2024-03-29T11:41:39.817 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.812+0000 7fa4ff7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fa4f8006d80 0x7fa4f80cbc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50236/0 (socket says 172.21.15.72:50236) 2024-03-29T11:41:39.818 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.813+0000 7fa4ff7fe640 1 -- 172.21.15.72:0/4087507894 learned_addr learned my addr 172.21.15.72:0/4087507894 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:41:39.818 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.813+0000 7fa4fe7fc640 1 -- 172.21.15.72:0/4087507894 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 8753626 0 0) 0x7fa4f8008470 con 0x7fa4f8006d80 2024-03-29T11:41:39.818 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.813+0000 7fa4fe7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:41:39.818 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.813+0000 7fa4fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:39.818 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.813+0000 7fa4fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:39.818 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.813+0000 7fa4fe7fc640 10 monclient(hunting): my global_id is 4464 2024-03-29T11:41:39.818 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.813+0000 7fa4fe7fc640 1 -- 172.21.15.72:0/4087507894 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4f0003520 con 0x7fa4f8006d80 2024-03-29T11:41:39.818 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.814+0000 7fa4fe7fc640 1 -- 172.21.15.72:0/4087507894 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3289778343 0 0) 0x7fa4f0003520 con 0x7fa4f8006d80 2024-03-29T11:41:39.818 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.814+0000 7fa4fe7fc640 1 -- 172.21.15.72:0/4087507894 >> v1:172.21.15.72:6790/0 conn(0x7fa4f80c9430 legacy=0x7fa4f80d74c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:39.820 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.817+0000 7fa4fe7fc640 1 -- 172.21.15.72:0/4087507894 >> v1:172.21.15.89:6789/0 conn(0x7fa4f80cc310 legacy=0x7fa4f80c8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:41:39.820 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.817+0000 7fa4fe7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:41:39.820 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.817+0000 7fa4fe7fc640 1 monclient: found mon.noname-a 2024-03-29T11:41:39.820 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.817+0000 7fa4fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:39.821 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.817+0000 7fa4fe7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.72:6789/0 2024-03-29T11:41:39.821 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.817+0000 7fa4fe7fc640 1 -- 172.21.15.72:0/4087507894 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f802c3f0 con 0x7fa4f8006d80 2024-03-29T11:41:39.821 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.817+0000 7fa4fe7fc640 10 monclient: _finish_auth 0 2024-03-29T11:41:39.821 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.817+0000 7fa4fe7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:41:39.821 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.817+0000 7fa4fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:39.821 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.817+0000 7fa4fe7fc640 1 -- 172.21.15.72:0/4087507894 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fa4ec001690 con 0x7fa4f8006d80 2024-03-29T11:41:39.821 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.817+0000 7fa4fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:39.821 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.817+0000 7fa4fe7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:41:39.821 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.817+0000 7fa4fe7fc640 10 monclient: dump: 2024-03-29T11:41:39.821 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:41:39.821 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:41:39.821 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:39.821 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:39.821 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:39.822 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:39.822 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:41:39.822 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:41:39.822 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:41:39.822 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:39.822 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.818+0000 7fa504fdc640 5 monclient: authenticate success, global_id 4464 2024-03-29T11:41:39.822 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.818+0000 7fa504fdc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:39.823 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.818+0000 7fa4fe7fc640 1 -- 172.21.15.72:0/4087507894 <== mon.0 v1:172.21.15.72:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4ec0017f0 con 0x7fa4f8006d80 2024-03-29T11:41:39.823 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.818+0000 7fa4fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:39.823 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.818+0000 7fa504fdc640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:39.823 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.818+0000 7fa504fdc640 4 set_mon_vals no callback set 2024-03-29T11:41:39.823 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.819+0000 7fa4fe7fc640 1 -- 172.21.15.72:0/4087507894 <== mon.0 v1:172.21.15.72:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fa4ec001c40 con 0x7fa4f8006d80 2024-03-29T11:41:39.823 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.819+0000 7fa4fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:39.823 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.819+0000 7fa4fe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:39.823 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.819+0000 7fa4fe7fc640 10 monclient: dump: 2024-03-29T11:41:39.824 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:41:39.824 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:41:39.824 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:39.824 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:39.824 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:39.824 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:39.824 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:41:39.824 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:41:39.824 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:41:39.824 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:39.824 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.819+0000 7fa504fdc640 10 monclient: shutdown 2024-03-29T11:41:39.824 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.819+0000 7fa504fdc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:39.824 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.819+0000 7fa504fdc640 1 -- 172.21.15.72:0/4087507894 >> v1:172.21.15.72:6789/0 conn(0x7fa4f8006d80 legacy=0x7fa4f80cbc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:39.824 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.820+0000 7fa504fdc640 1 -- 172.21.15.72:0/4087507894 shutdown_connections 2024-03-29T11:41:39.825 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.820+0000 7fa504fdc640 1 -- 172.21.15.72:0/4087507894 >> 172.21.15.72:0/4087507894 conn(0x7fa4f8005c00 msgr2=0x7fa4f8006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:39.825 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.821+0000 7fa504fdc640 1 -- 172.21.15.72:0/4087507894 shutdown_connections 2024-03-29T11:41:39.826 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.821+0000 7fa504fdc640 1 -- 172.21.15.72:0/4087507894 wait complete. 2024-03-29T11:41:39.826 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.822+0000 7fa504fdc640 10 monclient: build_initial_monmap 2024-03-29T11:41:39.826 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.822+0000 7fa504fdc640 10 monclient: monmap: 2024-03-29T11:41:39.826 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:41:39.826 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:39.826 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:41:39.826 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:41:39.826 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:39.826 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:39.826 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:41:39.827 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:41:39.827 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:41:39.827 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:39.827 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.822+0000 7fa504fdc640 1 librados: starting msgr at 2024-03-29T11:41:39.827 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.822+0000 7fa504fdc640 1 librados: starting objecter 2024-03-29T11:41:39.827 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.823+0000 7fa504fdc640 1 Processor -- start 2024-03-29T11:41:39.832 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.823+0000 7fa504fdc640 1 -- start start 2024-03-29T11:41:39.832 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.823+0000 7fa504fdc640 1 librados: setting wanted keys 2024-03-29T11:41:39.832 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.823+0000 7fa504fdc640 1 librados: calling monclient init 2024-03-29T11:41:39.832 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.823+0000 7fa504fdc640 10 monclient: init 2024-03-29T11:41:39.832 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.824+0000 7fa504fdc640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:39.833 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.824+0000 7fa504fdc640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:41:39.833 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.824+0000 7fa504fdc640 10 monclient(hunting): picked mon.noname-b con 0x7fa4f8006d80 addr v1:172.21.15.89:6789/0 2024-03-29T11:41:39.833 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.824+0000 7fa504fdc640 10 monclient(hunting): picked mon.noname-c con 0x7fa4f80c9430 addr v1:172.21.15.72:6790/0 2024-03-29T11:41:39.833 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.824+0000 7fa504fdc640 10 monclient(hunting): picked mon.noname-a con 0x7fa4f80cc310 addr v1:172.21.15.72:6789/0 2024-03-29T11:41:39.833 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.824+0000 7fa504fdc640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4f80cad90 con 0x7fa4f80cc310 2024-03-29T11:41:39.833 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.824+0000 7fa504fdc640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4f80cbb90 con 0x7fa4f8006d80 2024-03-29T11:41:39.833 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.824+0000 7fa504fdc640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4f802c8b0 con 0x7fa4f80c9430 2024-03-29T11:41:39.833 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.824+0000 7fa504fdc640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:39.833 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.824+0000 7fa504fdc640 10 monclient(hunting): authenticate will time out at 1259.112320s 2024-03-29T11:41:39.833 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.824+0000 7fa4fffff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fa4f80cc310 0x7fa4f816c3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50426/0 (socket says 172.21.15.72:50426) 2024-03-29T11:41:39.833 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.824+0000 7fa4fffff640 1 -- 172.21.15.72:0/329312029 learned_addr learned my addr 172.21.15.72:0/329312029 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:41:39.833 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.824+0000 7fa4ebfff640 10 client.?.objecter ms_handle_connect 0x7fa4f80cc310 2024-03-29T11:41:39.833 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.824+0000 7fa4ebfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:39.833 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.825+0000 7fa4ebfff640 1 -- 172.21.15.72:0/329312029 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4120394501 0 0) 0x7fa4f80cad90 con 0x7fa4f80cc310 2024-03-29T11:41:39.833 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.825+0000 7fa4ebfff640 10 monclient(hunting): _negotiate 2024-03-29T11:41:39.833 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.825+0000 7fa4ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:39.834 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.825+0000 7fa4ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:39.834 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.825+0000 7fa4ebfff640 10 monclient(hunting): my global_id is 4467 2024-03-29T11:41:39.834 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.825+0000 7fa4ebfff640 1 -- 172.21.15.72:0/329312029 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4e4004eb0 con 0x7fa4f80cc310 2024-03-29T11:41:39.834 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.826+0000 7fa4ebfff640 1 -- 172.21.15.72:0/329312029 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 67196184 0 0) 0x7fa4e4004eb0 con 0x7fa4f80cc310 2024-03-29T11:41:39.834 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.826+0000 7fa4ebfff640 1 -- 172.21.15.72:0/329312029 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4f80cad90 con 0x7fa4f80cc310 2024-03-29T11:41:39.834 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.826+0000 7fa4ebfff640 1 -- 172.21.15.72:0/329312029 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fa50004fdd0 con 0x7fa4f80cc310 2024-03-29T11:41:39.834 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.826+0000 7fa4ebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:39.834 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.826+0000 7fa4ebfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:41:39.834 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.826+0000 7fa4ebfff640 10 monclient(hunting): dump: 2024-03-29T11:41:39.834 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:41:39.834 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:41:39.834 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:39.834 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:39.834 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:39.834 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:39.835 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:41:39.835 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:41:39.835 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:41:39.835 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:39.835 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.826+0000 7fa4ebfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:39.835 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.828+0000 7fa4ebfff640 10 client.?.objecter ms_handle_connect 0x7fa4f8006d80 2024-03-29T11:41:39.835 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.828+0000 7fa4ebfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:39.835 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.828+0000 7fa4ebfff640 1 -- 172.21.15.72:0/329312029 <== mon.0 v1:172.21.15.72:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 977763303 0 0) 0x7fa4f80cad90 con 0x7fa4f80cc310 2024-03-29T11:41:39.835 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.828+0000 7fa4ebfff640 1 -- 172.21.15.72:0/329312029 >> v1:172.21.15.72:6790/0 conn(0x7fa4f80c9430 legacy=0x7fa4f80c9a00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:41:39.835 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.828+0000 7fa4ebfff640 1 -- 172.21.15.72:0/329312029 >> v1:172.21.15.89:6789/0 conn(0x7fa4f8006d80 legacy=0x7fa4f80c3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:39.835 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.828+0000 7fa4ebfff640 10 monclient: _finish_hunting 0 2024-03-29T11:41:39.835 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.828+0000 7fa4ebfff640 1 monclient: found mon.a 2024-03-29T11:41:39.835 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.828+0000 7fa4ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:39.836 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.828+0000 7fa4ebfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:41:39.836 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.828+0000 7fa4ebfff640 1 -- 172.21.15.72:0/329312029 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f802ca40 con 0x7fa4f80cc310 2024-03-29T11:41:39.836 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.828+0000 7fa4ebfff640 10 monclient: _finish_auth 0 2024-03-29T11:41:39.836 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.828+0000 7fa4ebfff640 10 monclient: _check_auth_tickets 2024-03-29T11:41:39.836 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.828+0000 7fa4ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:39.836 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.829+0000 7fa504fdc640 5 monclient: authenticate success, global_id 4467 2024-03-29T11:41:39.836 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.829+0000 7fa504fdc640 10 monclient: _renew_subs 2024-03-29T11:41:39.836 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.829+0000 7fa504fdc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:41:39.836 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.829+0000 7fa504fdc640 1 -- 172.21.15.72:0/329312029 --> v1:172.21.15.72:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4f80c3f90 con 0x7fa4f80cc310 2024-03-29T11:41:39.836 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.829+0000 7fa504fdc640 10 client.4467.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:39.836 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.829+0000 7fa504fdc640 10 monclient: _renew_subs 2024-03-29T11:41:39.836 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.829+0000 7fa504fdc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:41:39.836 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.829+0000 7fa504fdc640 1 -- 172.21.15.72:0/329312029 --> v1:172.21.15.72:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4f8036560 con 0x7fa4f80cc310 2024-03-29T11:41:39.836 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.829+0000 7fa504fdc640 1 librados: init done 2024-03-29T11:41:39.839 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.830+0000 7fa504fdc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:39.839 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.830+0000 7fa504fdc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:39.839 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.830+0000 7fa504fdc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:41:39.839 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.830+0000 7fa504fdc640 1 -- 172.21.15.72:0/329312029 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4f80c4290 con 0x7fa4f80cc310 2024-03-29T11:41:39.839 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.831+0000 7fa4ebfff640 1 -- 172.21.15.72:0/329312029 <== mon.0 v1:172.21.15.72:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa50004f400 con 0x7fa4f80cc310 2024-03-29T11:41:39.839 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.831+0000 7fa4ebfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:39.839 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.831+0000 7fa4ebfff640 1 -- 172.21.15.72:0/329312029 <== mon.0 v1:172.21.15.72:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fa500075710 con 0x7fa4f80cc310 2024-03-29T11:41:39.839 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.831+0000 7fa4ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:39.840 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.831+0000 7fa4ebfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:39.840 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.831+0000 7fa4ebfff640 10 monclient: dump: 2024-03-29T11:41:39.840 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:41:39.840 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:41:39.840 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:39.840 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:39.840 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:39.840 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:39.840 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:41:39.840 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:41:39.840 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:41:39.840 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:39.840 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.831+0000 7fa4fd7fa640 4 set_mon_vals no callback set 2024-03-29T11:41:39.840 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.832+0000 7fa4ebfff640 1 -- 172.21.15.72:0/329312029 <== mon.0 v1:172.21.15.72:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7fa500067bf0 con 0x7fa4f80cc310 2024-03-29T11:41:39.840 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.833+0000 7fa4ebfff640 1 -- 172.21.15.72:0/329312029 <== mon.0 v1:172.21.15.72:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7fa5000ff560 con 0x7fa4f80cc310 2024-03-29T11:41:39.841 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.833+0000 7fa4ebfff640 10 client.4467.objecter ms_dispatch 0x7fa500000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:39.841 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.833+0000 7fa4ebfff640 3 client.4467.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:39.841 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.833+0000 7fa4ebfff640 3 client.4467.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:39.841 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.833+0000 7fa4ff7fe640 10 monclient: get_auth_request con 0x7fa4e406db50 auth_method 0 2024-03-29T11:41:39.841 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.833+0000 7fa4ebfff640 20 client.4467.objecter dump_active .. 0 homeless 2024-03-29T11:41:39.841 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.833+0000 7fa4ebfff640 10 client.4467.objecter ms_handle_connect 0x7fa4e406db50 2024-03-29T11:41:39.842 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.839+0000 7fa4ebfff640 1 -- 172.21.15.72:0/329312029 <== mon.0 v1:172.21.15.72:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa5000cae60 con 0x7fa4f80cc310 2024-03-29T11:41:39.842 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.839+0000 7fa4ebfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:39.842 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:39.839+0000 7fa4ebfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:40.166 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:40.164+0000 7fa504fdc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:40.167 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:40.164+0000 7fa504fdc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:40.167 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:40.164+0000 7fa504fdc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:41:40.167 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:40.164+0000 7fa504fdc640 1 -- 172.21.15.72:0/329312029 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa4f80cbd20 con 0x7fa4f80cc310 2024-03-29T11:41:40.168 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:40.166+0000 7fa4ebfff640 1 -- 172.21.15.72:0/329312029 <== mon.0 v1:172.21.15.72:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7fa5000ccd90 con 0x7fa4f80cc310 2024-03-29T11:41:40.168 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:40.166+0000 7fa4ebfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:40.168 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:40.166+0000 7fa4ebfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:40.169 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:41:40.169 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:40.173 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:40.170+0000 7fa504fdc640 10 librados: watch_flush enter 2024-03-29T11:41:40.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:40.171+0000 7fa504fdc640 10 librados: watch_flush exit 2024-03-29T11:41:40.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:40.171+0000 7fa504fdc640 20 client.4467.objecter shutdown clearing up homeless session... 2024-03-29T11:41:40.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:40.171+0000 7fa504fdc640 10 client.4467.objecter successfully canceled tick 2024-03-29T11:41:40.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:40.172+0000 7fa504fdc640 1 -- 172.21.15.72:0/329312029 >> v1:172.21.15.72:6816/1578866102 conn(0x7fa4e406db50 legacy=0x7fa4e4070010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:40.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:40.172+0000 7fa504fdc640 10 monclient: shutdown 2024-03-29T11:41:40.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:40.172+0000 7fa504fdc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:40.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:40.172+0000 7fa504fdc640 1 -- 172.21.15.72:0/329312029 >> v1:172.21.15.72:6789/0 conn(0x7fa4f80cc310 legacy=0x7fa4f816c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:40.175 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:40.172+0000 7fa504fdc640 1 -- 172.21.15.72:0/329312029 shutdown_connections 2024-03-29T11:41:40.175 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:40.172+0000 7fa504fdc640 1 -- 172.21.15.72:0/329312029 >> 172.21.15.72:0/329312029 conn(0x7fa4f8005c00 msgr2=0x7fa4f80c8af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:40.175 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:40.172+0000 7fa504fdc640 1 -- 172.21.15.72:0/329312029 shutdown_connections 2024-03-29T11:41:40.175 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:40.172+0000 7fa504fdc640 1 -- 172.21.15.72:0/329312029 wait complete. 2024-03-29T11:41:40.175 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:40.173+0000 7fa504fdc640 1 librados: shutdown 2024-03-29T11:41:40.193 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:47.000 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:47.149 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.146+0000 7f3facd99640 10 monclient: get_monmap_and_config 2024-03-29T11:41:47.149 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.146+0000 7f3facd99640 10 monclient: build_initial_monmap 2024-03-29T11:41:47.149 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.146+0000 7f3facd99640 10 monclient: monmap: 2024-03-29T11:41:47.149 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:41:47.149 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:41:47.149 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:41:47.147263+0000 2024-03-29T11:41:47.149 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:41:47.147263+0000 2024-03-29T11:41:47.149 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:47.149 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:47.149 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:41:47.150 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:41:47.150 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:41:47.150 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:47.152 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.147+0000 7f3facd99640 1 Processor -- start 2024-03-29T11:41:47.152 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.147+0000 7f3facd99640 1 -- start start 2024-03-29T11:41:47.152 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.147+0000 7f3facd99640 10 monclient: init 2024-03-29T11:41:47.152 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.148+0000 7f3facd99640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:47.152 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.148+0000 7f3facd99640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:41:47.152 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.148+0000 7f3facd99640 10 monclient(hunting): picked mon.noname-a con 0x7f3fa80a9750 addr v1:172.21.15.72:6789/0 2024-03-29T11:41:47.152 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.148+0000 7f3facd99640 10 monclient(hunting): picked mon.noname-b con 0x7f3fa80aad20 addr v1:172.21.15.72:6790/0 2024-03-29T11:41:47.152 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.148+0000 7f3facd99640 10 monclient(hunting): picked mon.noname-c con 0x7f3fa80a9b30 addr v1:172.21.15.89:6789/0 2024-03-29T11:41:47.152 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.148+0000 7f3facd99640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fa80af4d0 con 0x7f3fa80a9750 2024-03-29T11:41:47.152 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.148+0000 7f3facd99640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fa80af7b0 con 0x7f3fa80a9b30 2024-03-29T11:41:47.152 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.148+0000 7f3facd99640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fa80acab0 con 0x7f3fa80aad20 2024-03-29T11:41:47.152 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.148+0000 7f3facd99640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:47.152 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.148+0000 7f3facd99640 10 monclient(hunting): authenticate will time out at 1266.436524s 2024-03-29T11:41:47.152 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.148+0000 7f3fa77fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f3fa80a9750 0x7f3fa80ac3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:55580/0 (socket says 172.21.15.72:55580) 2024-03-29T11:41:47.153 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.148+0000 7f3fa77fe640 1 -- 172.21.15.72:0/2935427437 learned_addr learned my addr 172.21.15.72:0/2935427437 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:41:47.153 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.149+0000 7f3fa67fc640 1 -- 172.21.15.72:0/2935427437 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1727469205 0 0) 0x7f3fa80af4d0 con 0x7f3fa80a9750 2024-03-29T11:41:47.153 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.149+0000 7f3fa67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:41:47.153 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.149+0000 7f3fa67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:47.153 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.149+0000 7f3fa67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:47.153 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.149+0000 7f3fa67fc640 10 monclient(hunting): my global_id is 4470 2024-03-29T11:41:47.153 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.149+0000 7f3fa67fc640 1 -- 172.21.15.72:0/2935427437 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f90003520 con 0x7f3fa80a9750 2024-03-29T11:41:47.153 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.149+0000 7f3fa67fc640 1 -- 172.21.15.72:0/2935427437 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3965943912 0 0) 0x7f3f90003520 con 0x7f3fa80a9750 2024-03-29T11:41:47.153 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.149+0000 7f3fa67fc640 1 -- 172.21.15.72:0/2935427437 >> v1:172.21.15.72:6790/0 conn(0x7f3fa80aad20 legacy=0x7f3fa8141a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:47.155 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.152+0000 7f3fa67fc640 1 -- 172.21.15.72:0/2935427437 >> v1:172.21.15.89:6789/0 conn(0x7f3fa80a9b30 legacy=0x7f3fa8149930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:41:47.155 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.152+0000 7f3fa67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:41:47.155 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.152+0000 7f3fa67fc640 1 monclient: found mon.noname-a 2024-03-29T11:41:47.156 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.152+0000 7f3fa67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:47.156 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.152+0000 7f3fa67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.72:6789/0 2024-03-29T11:41:47.156 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.152+0000 7f3fa67fc640 1 -- 172.21.15.72:0/2935427437 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa80aba30 con 0x7f3fa80a9750 2024-03-29T11:41:47.156 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.152+0000 7f3fa67fc640 10 monclient: _finish_auth 0 2024-03-29T11:41:47.156 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.152+0000 7f3fa67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:41:47.156 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.152+0000 7f3fa67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:47.156 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.152+0000 7f3facd99640 5 monclient: authenticate success, global_id 4470 2024-03-29T11:41:47.156 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.152+0000 7f3facd99640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:47.156 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.152+0000 7f3fa67fc640 1 -- 172.21.15.72:0/2935427437 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f3f9c001690 con 0x7f3fa80a9750 2024-03-29T11:41:47.156 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.152+0000 7f3fa67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:47.156 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.152+0000 7f3fa67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:41:47.156 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.152+0000 7f3fa67fc640 10 monclient: dump: 2024-03-29T11:41:47.156 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:41:47.156 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:41:47.156 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:47.156 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:47.157 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:47.157 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:47.157 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:41:47.157 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:41:47.157 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:41:47.157 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:47.157 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.152+0000 7f3facd99640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:47.159 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.156+0000 7f3fa67fc640 1 -- 172.21.15.72:0/2935427437 <== mon.0 v1:172.21.15.72:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f9c0017f0 con 0x7f3fa80a9750 2024-03-29T11:41:47.159 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.156+0000 7f3fa67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:47.159 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.156+0000 7f3facd99640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:47.159 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.156+0000 7f3facd99640 4 set_mon_vals no callback set 2024-03-29T11:41:47.160 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.157+0000 7f3fa67fc640 1 -- 172.21.15.72:0/2935427437 <== mon.0 v1:172.21.15.72:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f3f9c001c40 con 0x7f3fa80a9750 2024-03-29T11:41:47.160 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.157+0000 7f3fa67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:47.160 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.157+0000 7f3fa67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:47.160 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.157+0000 7f3fa67fc640 10 monclient: dump: 2024-03-29T11:41:47.160 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:41:47.160 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:41:47.160 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:47.160 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:47.160 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:47.160 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:47.160 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:41:47.160 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:41:47.161 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:41:47.161 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:47.161 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.157+0000 7f3facd99640 10 monclient: shutdown 2024-03-29T11:41:47.161 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.157+0000 7f3facd99640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:47.161 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.157+0000 7f3facd99640 1 -- 172.21.15.72:0/2935427437 >> v1:172.21.15.72:6789/0 conn(0x7f3fa80a9750 legacy=0x7f3fa80ac3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:47.163 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.160+0000 7f3facd99640 1 -- 172.21.15.72:0/2935427437 shutdown_connections 2024-03-29T11:41:47.163 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.160+0000 7f3facd99640 1 -- 172.21.15.72:0/2935427437 >> 172.21.15.72:0/2935427437 conn(0x7f3fa80673e0 msgr2=0x7f3fa80b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:47.163 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.160+0000 7f3facd99640 1 -- 172.21.15.72:0/2935427437 shutdown_connections 2024-03-29T11:41:47.163 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.160+0000 7f3facd99640 1 -- 172.21.15.72:0/2935427437 wait complete. 2024-03-29T11:41:47.175 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.161+0000 7f3facd99640 10 monclient: build_initial_monmap 2024-03-29T11:41:47.175 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.161+0000 7f3facd99640 10 monclient: monmap: 2024-03-29T11:41:47.176 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:41:47.176 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:47.176 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:41:47.176 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:41:47.176 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:47.176 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:47.176 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:41:47.176 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:41:47.176 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:41:47.176 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:47.176 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.162+0000 7f3facd99640 1 librados: starting msgr at 2024-03-29T11:41:47.176 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.162+0000 7f3facd99640 1 librados: starting objecter 2024-03-29T11:41:47.176 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.162+0000 7f3facd99640 1 Processor -- start 2024-03-29T11:41:47.176 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.162+0000 7f3facd99640 1 -- start start 2024-03-29T11:41:47.177 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.162+0000 7f3facd99640 1 librados: setting wanted keys 2024-03-29T11:41:47.177 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.162+0000 7f3facd99640 1 librados: calling monclient init 2024-03-29T11:41:47.177 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.162+0000 7f3facd99640 10 monclient: init 2024-03-29T11:41:47.177 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.163+0000 7f3facd99640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:47.177 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.163+0000 7f3facd99640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:41:47.177 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.163+0000 7f3facd99640 10 monclient(hunting): picked mon.noname-c con 0x7f3fa80a9750 addr v1:172.21.15.72:6790/0 2024-03-29T11:41:47.177 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.163+0000 7f3facd99640 10 monclient(hunting): picked mon.noname-a con 0x7f3fa80a9b30 addr v1:172.21.15.72:6789/0 2024-03-29T11:41:47.177 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.165+0000 7f3fa6ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f3fa80a9b30 0x7f3fa813ab10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:57982/0 (socket says 172.21.15.72:57982) 2024-03-29T11:41:47.177 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.165+0000 7f3fa6ffd640 1 -- 172.21.15.72:0/1365955718 learned_addr learned my addr 172.21.15.72:0/1365955718 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:41:47.177 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.165+0000 7f3facd99640 10 monclient(hunting): picked mon.noname-b con 0x7f3fa80aad20 addr v1:172.21.15.89:6789/0 2024-03-29T11:41:47.177 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.165+0000 7f3facd99640 1 -- 172.21.15.72:0/1365955718 --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fa81d3270 con 0x7f3fa80a9b30 2024-03-29T11:41:47.177 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.165+0000 7f3f8ffff640 10 client.?.objecter ms_handle_connect 0x7f3fa80a9b30 2024-03-29T11:41:47.177 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.165+0000 7f3f8ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:47.177 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.165+0000 7f3facd99640 1 -- 172.21.15.72:0/1365955718 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fa813b930 con 0x7f3fa80aad20 2024-03-29T11:41:47.177 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.165+0000 7f3facd99640 1 -- 172.21.15.72:0/1365955718 --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fa81d6640 con 0x7f3fa80a9750 2024-03-29T11:41:47.178 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.165+0000 7f3facd99640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:47.178 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.165+0000 7f3facd99640 10 monclient(hunting): authenticate will time out at 1266.453813s 2024-03-29T11:41:47.178 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.166+0000 7f3f8ffff640 10 client.?.objecter ms_handle_connect 0x7f3fa80aad20 2024-03-29T11:41:47.178 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.166+0000 7f3f8ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:47.178 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.166+0000 7f3f8ffff640 1 -- 172.21.15.72:0/1365955718 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3822148444 0 0) 0x7f3fa81d3270 con 0x7f3fa80a9b30 2024-03-29T11:41:47.178 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.166+0000 7f3f8ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:41:47.178 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.166+0000 7f3f8ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:47.178 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.166+0000 7f3f8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:47.178 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.166+0000 7f3f8ffff640 10 monclient(hunting): my global_id is 4473 2024-03-29T11:41:47.178 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.166+0000 7f3f8ffff640 1 -- 172.21.15.72:0/1365955718 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f94004eb0 con 0x7f3fa80a9b30 2024-03-29T11:41:47.178 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.166+0000 7f3f8ffff640 1 -- 172.21.15.72:0/1365955718 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 986129716 0 0) 0x7f3fa813b930 con 0x7f3fa80aad20 2024-03-29T11:41:47.178 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.166+0000 7f3f8ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:41:47.179 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.166+0000 7f3f8ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:47.179 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.166+0000 7f3f8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:47.179 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.166+0000 7f3f8ffff640 10 monclient(hunting): my global_id is 4435 2024-03-29T11:41:47.179 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.166+0000 7f3f8ffff640 1 -- 172.21.15.72:0/1365955718 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fa81d3270 con 0x7f3fa80aad20 2024-03-29T11:41:47.179 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.166+0000 7f3f8ffff640 1 -- 172.21.15.72:0/1365955718 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1540384830 0 0) 0x7f3f94004eb0 con 0x7f3fa80a9b30 2024-03-29T11:41:47.179 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.166+0000 7f3f8ffff640 1 -- 172.21.15.72:0/1365955718 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3fa813b930 con 0x7f3fa80a9b30 2024-03-29T11:41:47.179 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.166+0000 7f3f8ffff640 1 -- 172.21.15.72:0/1365955718 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f3f980017d0 con 0x7f3fa80a9b30 2024-03-29T11:41:47.179 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.166+0000 7f3f8ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:47.179 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.166+0000 7f3f8ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:41:47.179 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.166+0000 7f3f8ffff640 10 monclient(hunting): dump: 2024-03-29T11:41:47.179 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:41:47.179 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:41:47.179 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:47.179 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:47.179 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:47.180 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:47.180 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:41:47.180 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:41:47.180 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:41:47.180 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:47.180 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.166+0000 7f3f8ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:47.180 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.166+0000 7f3f8ffff640 1 -- 172.21.15.72:0/1365955718 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4034020963 0 0) 0x7f3fa81d3270 con 0x7f3fa80aad20 2024-03-29T11:41:47.180 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.166+0000 7f3f8ffff640 1 -- 172.21.15.72:0/1365955718 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f94004eb0 con 0x7f3fa80aad20 2024-03-29T11:41:47.180 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.167+0000 7f3f8ffff640 1 -- 172.21.15.72:0/1365955718 <== mon.0 v1:172.21.15.72:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 996609757 0 0) 0x7f3fa813b930 con 0x7f3fa80a9b30 2024-03-29T11:41:47.180 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.167+0000 7f3f8ffff640 1 -- 172.21.15.72:0/1365955718 >> v1:172.21.15.72:6790/0 conn(0x7f3fa80a9750 legacy=0x7f3fa81d2b60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:41:47.180 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.168+0000 7f3f8ffff640 1 -- 172.21.15.72:0/1365955718 >> v1:172.21.15.89:6789/0 conn(0x7f3fa80aad20 legacy=0x7f3fa813b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:47.180 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.169+0000 7f3f8ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:41:47.180 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.169+0000 7f3f8ffff640 1 monclient: found mon.a 2024-03-29T11:41:47.180 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.169+0000 7f3f8ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:47.180 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.169+0000 7f3f8ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:41:47.181 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.169+0000 7f3f8ffff640 1 -- 172.21.15.72:0/1365955718 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fa81d6060 con 0x7f3fa80a9b30 2024-03-29T11:41:47.181 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.169+0000 7f3f8ffff640 10 monclient: _finish_auth 0 2024-03-29T11:41:47.181 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.169+0000 7f3f8ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:41:47.181 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.169+0000 7f3f8ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:47.181 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.169+0000 7f3facd99640 5 monclient: authenticate success, global_id 4473 2024-03-29T11:41:47.181 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.169+0000 7f3facd99640 10 monclient: _renew_subs 2024-03-29T11:41:47.181 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.169+0000 7f3facd99640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:41:47.181 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.169+0000 7f3facd99640 1 -- 172.21.15.72:0/1365955718 --> v1:172.21.15.72:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fa81d62b0 con 0x7f3fa80a9b30 2024-03-29T11:41:47.181 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.169+0000 7f3facd99640 10 client.4473.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:47.181 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.169+0000 7f3facd99640 10 monclient: _renew_subs 2024-03-29T11:41:47.181 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.169+0000 7f3facd99640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:41:47.181 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.169+0000 7f3facd99640 1 -- 172.21.15.72:0/1365955718 --> v1:172.21.15.72:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fa813bd50 con 0x7f3fa80a9b30 2024-03-29T11:41:47.181 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.169+0000 7f3facd99640 1 librados: init done 2024-03-29T11:41:47.181 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.170+0000 7f3f8ffff640 1 -- 172.21.15.72:0/1365955718 <== mon.0 v1:172.21.15.72:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f98001930 con 0x7f3fa80a9b30 2024-03-29T11:41:47.182 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.170+0000 7f3f8ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:47.182 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.170+0000 7f3f8ffff640 1 -- 172.21.15.72:0/1365955718 <== mon.0 v1:172.21.15.72:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f3f98004a00 con 0x7f3fa80a9b30 2024-03-29T11:41:47.182 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.170+0000 7f3f8ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:47.182 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.170+0000 7f3f8ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:47.182 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.170+0000 7f3f8ffff640 10 monclient: dump: 2024-03-29T11:41:47.182 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:41:47.182 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:41:47.182 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:47.182 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:47.182 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:47.182 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:47.182 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:41:47.182 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:41:47.182 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:41:47.182 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:47.183 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.170+0000 7f3fa57fa640 4 set_mon_vals no callback set 2024-03-29T11:41:47.183 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.170+0000 7f3facd99640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:47.183 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.171+0000 7f3facd99640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:47.183 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.171+0000 7f3facd99640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:41:47.183 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.171+0000 7f3facd99640 1 -- 172.21.15.72:0/1365955718 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fa81d64a0 con 0x7f3fa80a9b30 2024-03-29T11:41:47.183 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.171+0000 7f3f8ffff640 1 -- 172.21.15.72:0/1365955718 <== mon.0 v1:172.21.15.72:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f3f9801a9d0 con 0x7f3fa80a9b30 2024-03-29T11:41:47.183 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.172+0000 7f3f8ffff640 1 -- 172.21.15.72:0/1365955718 <== mon.0 v1:172.21.15.72:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f3f98084960 con 0x7f3fa80a9b30 2024-03-29T11:41:47.183 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.172+0000 7f3f8ffff640 10 client.4473.objecter ms_dispatch 0x7f3fa8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:47.183 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.172+0000 7f3f8ffff640 3 client.4473.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:47.183 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.172+0000 7f3f8ffff640 3 client.4473.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:47.183 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.172+0000 7f3f8ffff640 20 client.4473.objecter dump_active .. 0 homeless 2024-03-29T11:41:47.183 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.172+0000 7f3fa77fe640 10 monclient: get_auth_request con 0x7f3f9406dfc0 auth_method 0 2024-03-29T11:41:47.183 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.173+0000 7f3f8ffff640 10 client.4473.objecter ms_handle_connect 0x7f3f9406dfc0 2024-03-29T11:41:47.184 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.179+0000 7f3f8ffff640 1 -- 172.21.15.72:0/1365955718 <== mon.0 v1:172.21.15.72:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3fa81d64a0 con 0x7f3fa80a9b30 2024-03-29T11:41:47.184 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.179+0000 7f3f8ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:47.184 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.179+0000 7f3f8ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:47.510 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.507+0000 7f3facd99640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:47.510 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.507+0000 7f3facd99640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:47.510 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.507+0000 7f3facd99640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:41:47.510 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.507+0000 7f3facd99640 1 -- 172.21.15.72:0/1365955718 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3fa81d6440 con 0x7f3fa80a9b30 2024-03-29T11:41:47.511 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.508+0000 7f3f8ffff640 1 -- 172.21.15.72:0/1365955718 <== mon.0 v1:172.21.15.72:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f3f98050070 con 0x7f3fa80a9b30 2024-03-29T11:41:47.511 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.508+0000 7f3f8ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:47.511 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.508+0000 7f3f8ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:47.511 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:41:47.511 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:47.514 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.511+0000 7f3facd99640 10 librados: watch_flush enter 2024-03-29T11:41:47.514 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.511+0000 7f3facd99640 10 librados: watch_flush exit 2024-03-29T11:41:47.514 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.511+0000 7f3facd99640 20 client.4473.objecter shutdown clearing up homeless session... 2024-03-29T11:41:47.514 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.511+0000 7f3facd99640 10 client.4473.objecter successfully canceled tick 2024-03-29T11:41:47.514 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.512+0000 7f3facd99640 1 -- 172.21.15.72:0/1365955718 >> v1:172.21.15.72:6816/1578866102 conn(0x7f3f9406dfc0 legacy=0x7f3f94070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:47.515 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.512+0000 7f3facd99640 10 monclient: shutdown 2024-03-29T11:41:47.515 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.512+0000 7f3facd99640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:47.515 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.512+0000 7f3facd99640 1 -- 172.21.15.72:0/1365955718 >> v1:172.21.15.72:6789/0 conn(0x7f3fa80a9b30 legacy=0x7f3fa813ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:47.515 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.512+0000 7f3facd99640 1 -- 172.21.15.72:0/1365955718 shutdown_connections 2024-03-29T11:41:47.515 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.512+0000 7f3facd99640 1 -- 172.21.15.72:0/1365955718 >> 172.21.15.72:0/1365955718 conn(0x7f3fa80673e0 msgr2=0x7f3fa80b1410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:47.518 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.515+0000 7f3facd99640 1 -- 172.21.15.72:0/1365955718 shutdown_connections 2024-03-29T11:41:47.518 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.515+0000 7f3facd99640 1 -- 172.21.15.72:0/1365955718 wait complete. 2024-03-29T11:41:47.518 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:47.515+0000 7f3facd99640 1 librados: shutdown 2024-03-29T11:41:47.538 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:54.345 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:54.493 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.490+0000 7f80f31a8640 10 monclient: get_monmap_and_config 2024-03-29T11:41:54.493 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.490+0000 7f80f31a8640 10 monclient: build_initial_monmap 2024-03-29T11:41:54.493 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.490+0000 7f80f31a8640 10 monclient: monmap: 2024-03-29T11:41:54.493 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:41:54.493 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:41:54.493 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:41:54.491406+0000 2024-03-29T11:41:54.494 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:41:54.491406+0000 2024-03-29T11:41:54.494 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:54.494 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:54.494 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:41:54.494 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:41:54.494 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:41:54.494 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:54.494 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.491+0000 7f80f31a8640 1 Processor -- start 2024-03-29T11:41:54.495 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.492+0000 7f80f31a8640 1 -- start start 2024-03-29T11:41:54.495 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.492+0000 7f80f31a8640 10 monclient: init 2024-03-29T11:41:54.495 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.492+0000 7f80f31a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:54.495 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.492+0000 7f80f31a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:41:54.495 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.492+0000 7f80f31a8640 10 monclient(hunting): picked mon.noname-c con 0x7f80ec0b0770 addr v1:172.21.15.89:6789/0 2024-03-29T11:41:54.495 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.492+0000 7f80f31a8640 10 monclient(hunting): picked mon.noname-b con 0x7f80ec14a700 addr v1:172.21.15.72:6790/0 2024-03-29T11:41:54.495 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.492+0000 7f80f31a8640 10 monclient(hunting): picked mon.noname-a con 0x7f80ec0ac5a0 addr v1:172.21.15.72:6789/0 2024-03-29T11:41:54.496 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.492+0000 7f80f31a8640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80ec0aba70 con 0x7f80ec0ac5a0 2024-03-29T11:41:54.496 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.492+0000 7f80f31a8640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80ec0aecb0 con 0x7f80ec0b0770 2024-03-29T11:41:54.496 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.492+0000 7f80f31a8640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80ec0af150 con 0x7f80ec14a700 2024-03-29T11:41:54.496 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.492+0000 7f80f31a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:54.496 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.492+0000 7f80f31a8640 10 monclient(hunting): authenticate will time out at 1273.780993s 2024-03-29T11:41:54.496 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.493+0000 7f80ebfff640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f80ec14a700 0x7f80ec0abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:38440/0 (socket says 172.21.15.72:38440) 2024-03-29T11:41:54.496 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.493+0000 7f80ebfff640 1 -- 172.21.15.72:0/2754513814 learned_addr learned my addr 172.21.15.72:0/2754513814 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:41:54.497 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.493+0000 7f80eb7fe640 1 -- 172.21.15.72:0/2754513814 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1505794169 0 0) 0x7f80ec0af150 con 0x7f80ec14a700 2024-03-29T11:41:54.497 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.493+0000 7f80eb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:41:54.497 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.493+0000 7f80eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:54.497 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.493+0000 7f80eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:54.497 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.493+0000 7f80eb7fe640 10 monclient(hunting): my global_id is 4478 2024-03-29T11:41:54.497 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.493+0000 7f80eb7fe640 1 -- 172.21.15.72:0/2754513814 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80d4003520 con 0x7f80ec14a700 2024-03-29T11:41:54.497 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.494+0000 7f80eb7fe640 1 -- 172.21.15.72:0/2754513814 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 452685578 0 0) 0x7f80d4003520 con 0x7f80ec14a700 2024-03-29T11:41:54.497 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.494+0000 7f80eb7fe640 1 -- 172.21.15.72:0/2754513814 >> v1:172.21.15.89:6789/0 conn(0x7f80ec0b0770 legacy=0x7f80ec149ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:54.499 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.496+0000 7f80eb7fe640 1 -- 172.21.15.72:0/2754513814 >> v1:172.21.15.72:6789/0 conn(0x7f80ec0ac5a0 legacy=0x7f80ec151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:41:54.501 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.496+0000 7f80eb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:41:54.501 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.496+0000 7f80eb7fe640 1 monclient: found mon.noname-b 2024-03-29T11:41:54.501 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.496+0000 7f80eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:54.501 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.496+0000 7f80eb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.72:6790/0 2024-03-29T11:41:54.501 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.496+0000 7f80eb7fe640 1 -- 172.21.15.72:0/2754513814 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80ec0ac9d0 con 0x7f80ec14a700 2024-03-29T11:41:54.502 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.496+0000 7f80eb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:41:54.502 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.496+0000 7f80eb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:41:54.502 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.496+0000 7f80eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:54.502 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.496+0000 7f80eb7fe640 1 -- 172.21.15.72:0/2754513814 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f80e0001690 con 0x7f80ec14a700 2024-03-29T11:41:54.502 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.496+0000 7f80eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:54.502 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.496+0000 7f80eb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:54.502 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.496+0000 7f80eb7fe640 10 monclient: dump: 2024-03-29T11:41:54.502 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:41:54.502 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:41:54.502 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:54.502 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:54.502 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:54.502 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:54.502 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:41:54.502 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:41:54.503 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:41:54.503 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:54.503 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.497+0000 7f80eb7fe640 1 -- 172.21.15.72:0/2754513814 <== mon.2 v1:172.21.15.72:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80e0001860 con 0x7f80ec14a700 2024-03-29T11:41:54.503 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.497+0000 7f80eb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:54.503 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.497+0000 7f80eb7fe640 1 -- 172.21.15.72:0/2754513814 <== mon.2 v1:172.21.15.72:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f80e0001cb0 con 0x7f80ec14a700 2024-03-29T11:41:54.503 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.497+0000 7f80eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:54.503 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.497+0000 7f80eb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:54.503 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.497+0000 7f80eb7fe640 10 monclient: dump: 2024-03-29T11:41:54.503 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:41:54.503 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:41:54.503 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:54.503 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:54.503 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:54.503 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:54.503 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:41:54.503 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:41:54.504 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:41:54.504 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:54.504 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.498+0000 7f80f31a8640 5 monclient: authenticate success, global_id 4478 2024-03-29T11:41:54.504 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.498+0000 7f80f31a8640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:54.504 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.498+0000 7f80f31a8640 4 set_mon_vals no callback set 2024-03-29T11:41:54.504 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.499+0000 7f80f31a8640 10 monclient: shutdown 2024-03-29T11:41:54.504 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.499+0000 7f80f31a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:54.504 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.499+0000 7f80f31a8640 1 -- 172.21.15.72:0/2754513814 >> v1:172.21.15.72:6790/0 conn(0x7f80ec14a700 legacy=0x7f80ec0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:54.505 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.499+0000 7f80f31a8640 1 -- 172.21.15.72:0/2754513814 shutdown_connections 2024-03-29T11:41:54.505 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.499+0000 7f80f31a8640 1 -- 172.21.15.72:0/2754513814 >> 172.21.15.72:0/2754513814 conn(0x7f80ec06b660 msgr2=0x7f80ec06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:54.506 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.500+0000 7f80f31a8640 1 -- 172.21.15.72:0/2754513814 shutdown_connections 2024-03-29T11:41:54.506 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.500+0000 7f80f31a8640 1 -- 172.21.15.72:0/2754513814 wait complete. 2024-03-29T11:41:54.506 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.500+0000 7f80f31a8640 10 monclient: build_initial_monmap 2024-03-29T11:41:54.506 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.501+0000 7f80f31a8640 10 monclient: monmap: 2024-03-29T11:41:54.506 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:41:54.506 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:54.506 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:41:54.506 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:41:54.506 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:54.506 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:54.506 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:41:54.506 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:41:54.506 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:41:54.506 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:54.506 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.501+0000 7f80f31a8640 1 librados: starting msgr at 2024-03-29T11:41:54.506 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.501+0000 7f80f31a8640 1 librados: starting objecter 2024-03-29T11:41:54.507 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.501+0000 7f80f31a8640 1 Processor -- start 2024-03-29T11:41:54.507 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.501+0000 7f80f31a8640 1 -- start start 2024-03-29T11:41:54.507 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.501+0000 7f80f31a8640 1 librados: setting wanted keys 2024-03-29T11:41:54.507 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.501+0000 7f80f31a8640 1 librados: calling monclient init 2024-03-29T11:41:54.507 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.501+0000 7f80f31a8640 10 monclient: init 2024-03-29T11:41:54.507 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.502+0000 7f80f31a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:54.507 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.502+0000 7f80f31a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:41:54.507 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.502+0000 7f80f31a8640 10 monclient(hunting): picked mon.noname-b con 0x7f80ec0ac5a0 addr v1:172.21.15.89:6789/0 2024-03-29T11:41:54.514 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.503+0000 7f80f31a8640 10 monclient(hunting): picked mon.noname-a con 0x7f80ec0b0770 addr v1:172.21.15.72:6789/0 2024-03-29T11:41:54.514 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.503+0000 7f80f31a8640 10 monclient(hunting): picked mon.noname-c con 0x7f80ec11e460 addr v1:172.21.15.72:6790/0 2024-03-29T11:41:54.514 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.503+0000 7f80f31a8640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80ec11de20 con 0x7f80ec0b0770 2024-03-29T11:41:54.514 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.503+0000 7f80f31a8640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80ec11d070 con 0x7f80ec0ac5a0 2024-03-29T11:41:54.514 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.503+0000 7f80f31a8640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80ec11d870 con 0x7f80ec11e460 2024-03-29T11:41:54.514 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.503+0000 7f80f31a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:54.514 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.503+0000 7f80f31a8640 10 monclient(hunting): authenticate will time out at 1273.791513s 2024-03-29T11:41:54.515 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.503+0000 7f80f1745640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f80ec11e460 0x7f80ec1ef850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:39538/0 (socket says 172.21.15.72:39538) 2024-03-29T11:41:54.515 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.503+0000 7f80f1745640 1 -- 172.21.15.72:0/1302408642 learned_addr learned my addr 172.21.15.72:0/1302408642 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:41:54.515 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.506+0000 7f80e97fa640 10 client.?.objecter ms_handle_connect 0x7f80ec11e460 2024-03-29T11:41:54.515 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.506+0000 7f80e97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:54.515 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.506+0000 7f80e97fa640 1 -- 172.21.15.72:0/1302408642 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3065652399 0 0) 0x7f80ec11d870 con 0x7f80ec11e460 2024-03-29T11:41:54.515 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.506+0000 7f80e97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:41:54.515 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.506+0000 7f80e97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:54.515 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.506+0000 7f80e97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:54.515 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.506+0000 7f80e97fa640 10 monclient(hunting): my global_id is 4481 2024-03-29T11:41:54.515 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.506+0000 7f80e97fa640 1 -- 172.21.15.72:0/1302408642 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80d8004ee0 con 0x7f80ec11e460 2024-03-29T11:41:54.515 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.507+0000 7f80e97fa640 1 -- 172.21.15.72:0/1302408642 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2660002167 0 0) 0x7f80d8004ee0 con 0x7f80ec11e460 2024-03-29T11:41:54.515 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.507+0000 7f80e97fa640 1 -- 172.21.15.72:0/1302408642 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80ec11d870 con 0x7f80ec11e460 2024-03-29T11:41:54.515 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.507+0000 7f80e97fa640 1 -- 172.21.15.72:0/1302408642 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f80dc001910 con 0x7f80ec11e460 2024-03-29T11:41:54.515 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.507+0000 7f80e97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:54.516 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.507+0000 7f80e97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:54.516 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.507+0000 7f80e97fa640 10 monclient(hunting): dump: 2024-03-29T11:41:54.516 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:41:54.516 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:41:54.516 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:54.516 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:54.516 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:54.516 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:54.516 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:41:54.516 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:41:54.516 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:41:54.516 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:54.516 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.507+0000 7f80e97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:54.516 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.508+0000 7f80e97fa640 10 client.?.objecter ms_handle_connect 0x7f80ec0ac5a0 2024-03-29T11:41:54.516 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.508+0000 7f80e97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:54.516 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.508+0000 7f80e97fa640 1 -- 172.21.15.72:0/1302408642 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2240110139 0 0) 0x7f80ec11d070 con 0x7f80ec0ac5a0 2024-03-29T11:41:54.517 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.508+0000 7f80e97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:41:54.517 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.509+0000 7f80e97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:54.517 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.509+0000 7f80e97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:54.517 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.509+0000 7f80e97fa640 10 monclient(hunting): my global_id is 4438 2024-03-29T11:41:54.517 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.509+0000 7f80e97fa640 1 -- 172.21.15.72:0/1302408642 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80d8004ee0 con 0x7f80ec0ac5a0 2024-03-29T11:41:54.517 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.509+0000 7f80e97fa640 10 client.?.objecter ms_handle_connect 0x7f80ec0b0770 2024-03-29T11:41:54.517 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.509+0000 7f80e97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:54.517 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.509+0000 7f80e97fa640 1 -- 172.21.15.72:0/1302408642 <== mon.2 v1:172.21.15.72:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3910462995 0 0) 0x7f80ec11d870 con 0x7f80ec11e460 2024-03-29T11:41:54.517 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.510+0000 7f80e97fa640 1 -- 172.21.15.72:0/1302408642 >> v1:172.21.15.89:6789/0 conn(0x7f80ec0ac5a0 legacy=0x7f80ec147e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:54.517 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.510+0000 7f80e97fa640 1 -- 172.21.15.72:0/1302408642 >> v1:172.21.15.72:6789/0 conn(0x7f80ec0b0770 legacy=0x7f80ec1e7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:54.517 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.510+0000 7f80e97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:41:54.517 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.510+0000 7f80e97fa640 1 monclient: found mon.c 2024-03-29T11:41:54.517 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.510+0000 7f80e97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:54.517 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.510+0000 7f80e97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:41:54.517 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.510+0000 7f80e97fa640 1 -- 172.21.15.72:0/1302408642 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80ec11da00 con 0x7f80ec11e460 2024-03-29T11:41:54.517 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.510+0000 7f80e97fa640 10 monclient: _finish_auth 0 2024-03-29T11:41:54.518 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.510+0000 7f80e97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:41:54.518 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.510+0000 7f80e97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:54.518 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.510+0000 7f80e97fa640 1 -- 172.21.15.72:0/1302408642 <== mon.2 v1:172.21.15.72:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80dc003610 con 0x7f80ec11e460 2024-03-29T11:41:54.518 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.511+0000 7f80f31a8640 5 monclient: authenticate success, global_id 4481 2024-03-29T11:41:54.518 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.511+0000 7f80f31a8640 10 monclient: _renew_subs 2024-03-29T11:41:54.518 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.511+0000 7f80f31a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:41:54.518 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.511+0000 7f80f31a8640 1 -- 172.21.15.72:0/1302408642 --> v1:172.21.15.72:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80ec0706e0 con 0x7f80ec11e460 2024-03-29T11:41:54.518 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.511+0000 7f80f31a8640 10 client.4481.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:54.521 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.511+0000 7f80f31a8640 10 monclient: _renew_subs 2024-03-29T11:41:54.521 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.511+0000 7f80f31a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:41:54.521 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.511+0000 7f80f31a8640 1 -- 172.21.15.72:0/1302408642 --> v1:172.21.15.72:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f80ec148570 con 0x7f80ec11e460 2024-03-29T11:41:54.522 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.511+0000 7f80f31a8640 1 librados: init done 2024-03-29T11:41:54.522 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.511+0000 7f80e97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:54.522 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.511+0000 7f80e97fa640 1 -- 172.21.15.72:0/1302408642 <== mon.2 v1:172.21.15.72:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f80dc003a60 con 0x7f80ec11e460 2024-03-29T11:41:54.522 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.511+0000 7f80e97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:54.522 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.511+0000 7f80ea7fc640 4 set_mon_vals no callback set 2024-03-29T11:41:54.523 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.511+0000 7f80e97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:54.523 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.512+0000 7f80e97fa640 10 monclient: dump: 2024-03-29T11:41:54.524 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:41:54.524 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:41:54.524 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:54.524 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:41:54.524 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:54.524 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:41:54.524 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:41:54.524 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:41:54.524 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:41:54.524 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:41:54.524 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.513+0000 7f80f31a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:54.524 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.514+0000 7f80e97fa640 1 -- 172.21.15.72:0/1302408642 <== mon.2 v1:172.21.15.72:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f80dc003c40 con 0x7f80ec11e460 2024-03-29T11:41:54.524 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.515+0000 7f80e97fa640 1 -- 172.21.15.72:0/1302408642 <== mon.2 v1:172.21.15.72:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f80dc084b60 con 0x7f80ec11e460 2024-03-29T11:41:54.524 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.515+0000 7f80f31a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:54.524 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.515+0000 7f80f31a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:41:54.524 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.515+0000 7f80f31a8640 1 -- 172.21.15.72:0/1302408642 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80cc005080 con 0x7f80ec11e460 2024-03-29T11:41:54.525 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.516+0000 7f80e97fa640 10 client.4481.objecter ms_dispatch 0x7f80ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:54.525 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.516+0000 7f80e97fa640 3 client.4481.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:54.525 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.516+0000 7f80e97fa640 3 client.4481.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:54.525 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.516+0000 7f80e97fa640 20 client.4481.objecter dump_active .. 0 homeless 2024-03-29T11:41:54.525 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.516+0000 7f80f0f44640 10 monclient: get_auth_request con 0x7f80d806dce0 auth_method 0 2024-03-29T11:41:54.525 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.517+0000 7f80e97fa640 10 client.4481.objecter ms_handle_connect 0x7f80d806dce0 2024-03-29T11:41:54.527 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.524+0000 7f80e97fa640 1 -- 172.21.15.72:0/1302408642 <== mon.2 v1:172.21.15.72:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f80dc050440 con 0x7f80ec11e460 2024-03-29T11:41:54.527 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.524+0000 7f80e97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:54.527 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.524+0000 7f80e97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:54.857 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.855+0000 7f80f31a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:54.857 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.855+0000 7f80f31a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:54.858 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.855+0000 7f80f31a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:41:54.858 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.855+0000 7f80f31a8640 1 -- 172.21.15.72:0/1302408642 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80cc005330 con 0x7f80ec11e460 2024-03-29T11:41:54.858 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.856+0000 7f80e97fa640 1 -- 172.21.15.72:0/1302408642 <== mon.2 v1:172.21.15.72:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f80dc052370 con 0x7f80ec11e460 2024-03-29T11:41:54.858 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.856+0000 7f80e97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:54.858 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.856+0000 7f80e97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:54.859 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:41:54.859 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:54.863 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.860+0000 7f80f31a8640 10 librados: watch_flush enter 2024-03-29T11:41:54.864 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.861+0000 7f80f31a8640 10 librados: watch_flush exit 2024-03-29T11:41:54.864 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.861+0000 7f80f31a8640 20 client.4481.objecter shutdown clearing up homeless session... 2024-03-29T11:41:54.864 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.861+0000 7f80f31a8640 10 client.4481.objecter successfully canceled tick 2024-03-29T11:41:54.864 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.861+0000 7f80f31a8640 1 -- 172.21.15.72:0/1302408642 >> v1:172.21.15.72:6816/1578866102 conn(0x7f80d806dce0 legacy=0x7f80d80701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:54.864 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.861+0000 7f80f31a8640 10 monclient: shutdown 2024-03-29T11:41:54.864 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.861+0000 7f80f31a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:54.864 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.861+0000 7f80f31a8640 1 -- 172.21.15.72:0/1302408642 >> v1:172.21.15.72:6790/0 conn(0x7f80ec11e460 legacy=0x7f80ec1ef850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:54.866 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.863+0000 7f80f31a8640 1 -- 172.21.15.72:0/1302408642 shutdown_connections 2024-03-29T11:41:54.866 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.863+0000 7f80f31a8640 1 -- 172.21.15.72:0/1302408642 >> 172.21.15.72:0/1302408642 conn(0x7f80ec06b660 msgr2=0x7f80ec149480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:54.866 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.863+0000 7f80f31a8640 1 -- 172.21.15.72:0/1302408642 shutdown_connections 2024-03-29T11:41:54.866 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.863+0000 7f80f31a8640 1 -- 172.21.15.72:0/1302408642 wait complete. 2024-03-29T11:41:54.868 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:41:54.865+0000 7f80f31a8640 1 librados: shutdown 2024-03-29T11:41:54.884 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:01.691 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:01.838 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.835+0000 7f1d09a94640 10 monclient: get_monmap_and_config 2024-03-29T11:42:01.838 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.835+0000 7f1d09a94640 10 monclient: build_initial_monmap 2024-03-29T11:42:01.838 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.835+0000 7f1d09a94640 10 monclient: monmap: 2024-03-29T11:42:01.839 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:42:01.839 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:01.839 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:42:01.836578+0000 2024-03-29T11:42:01.839 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:42:01.836578+0000 2024-03-29T11:42:01.839 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:01.839 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:01.839 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:42:01.839 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:42:01.839 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:42:01.839 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:01.842 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.839+0000 7f1d09a94640 1 Processor -- start 2024-03-29T11:42:01.842 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.839+0000 7f1d09a94640 1 -- start start 2024-03-29T11:42:01.842 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.839+0000 7f1d09a94640 10 monclient: init 2024-03-29T11:42:01.843 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.840+0000 7f1d09a94640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:01.843 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.840+0000 7f1d09a94640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:42:01.843 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.840+0000 7f1d09a94640 10 monclient(hunting): picked mon.noname-b con 0x7f1cfc006a70 addr v1:172.21.15.72:6790/0 2024-03-29T11:42:01.847 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.840+0000 7f1d09a94640 10 monclient(hunting): picked mon.noname-c con 0x7f1cfc033060 addr v1:172.21.15.89:6789/0 2024-03-29T11:42:01.847 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.843+0000 7f1d08a92640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f1cfc006a70 0x7f1cfc0c7a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:47456/0 (socket says 172.21.15.72:47456) 2024-03-29T11:42:01.847 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.844+0000 7f1d09a94640 10 monclient(hunting): picked mon.noname-a con 0x7f1cfc0c1e70 addr v1:172.21.15.72:6789/0 2024-03-29T11:42:01.847 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.844+0000 7f1d08a92640 1 -- 172.21.15.72:0/3728965688 learned_addr learned my addr 172.21.15.72:0/3728965688 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:42:01.847 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.844+0000 7f1d09a94640 1 -- 172.21.15.72:0/3728965688 --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cfc02cfb0 con 0x7f1cfc0c1e70 2024-03-29T11:42:01.847 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.844+0000 7f1d09a94640 1 -- 172.21.15.72:0/3728965688 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cfc02ae70 con 0x7f1cfc033060 2024-03-29T11:42:01.847 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.844+0000 7f1d09a94640 1 -- 172.21.15.72:0/3728965688 --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cfc0c8110 con 0x7f1cfc006a70 2024-03-29T11:42:01.847 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.844+0000 7f1d09a94640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:01.847 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.844+0000 7f1d09a94640 10 monclient(hunting): authenticate will time out at 1281.132526s 2024-03-29T11:42:01.848 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.844+0000 7f1d037fe640 1 -- 172.21.15.72:0/3728965688 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4079819294 0 0) 0x7f1cfc0c8110 con 0x7f1cfc006a70 2024-03-29T11:42:01.848 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.844+0000 7f1d037fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:01.848 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.844+0000 7f1d037fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:01.848 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.844+0000 7f1d037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:01.848 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.845+0000 7f1d037fe640 10 monclient(hunting): my global_id is 4484 2024-03-29T11:42:01.848 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.845+0000 7f1d037fe640 1 -- 172.21.15.72:0/3728965688 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cf4003520 con 0x7f1cfc006a70 2024-03-29T11:42:01.848 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.845+0000 7f1d037fe640 1 -- 172.21.15.72:0/3728965688 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1825503353 0 0) 0x7f1cf4003520 con 0x7f1cfc006a70 2024-03-29T11:42:01.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.845+0000 7f1d037fe640 1 -- 172.21.15.72:0/3728965688 >> v1:172.21.15.89:6789/0 conn(0x7f1cfc033060 legacy=0x7f1cfc0c1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:42:01.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.845+0000 7f1d037fe640 1 -- 172.21.15.72:0/3728965688 >> v1:172.21.15.72:6789/0 conn(0x7f1cfc0c1e70 legacy=0x7f1cfc0cfda0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:42:01.850 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.847+0000 7f1d037fe640 10 monclient: _finish_hunting 0 2024-03-29T11:42:01.850 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.847+0000 7f1d037fe640 1 monclient: found mon.noname-b 2024-03-29T11:42:01.850 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.847+0000 7f1d037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:01.850 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.847+0000 7f1d037fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.72:6790/0 2024-03-29T11:42:01.850 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.847+0000 7f1d037fe640 1 -- 172.21.15.72:0/3728965688 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cfc02d200 con 0x7f1cfc006a70 2024-03-29T11:42:01.850 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.847+0000 7f1d037fe640 10 monclient: _finish_auth 0 2024-03-29T11:42:01.850 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.847+0000 7f1d037fe640 10 monclient: _check_auth_tickets 2024-03-29T11:42:01.850 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.847+0000 7f1d037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:01.850 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.847+0000 7f1d037fe640 1 -- 172.21.15.72:0/3728965688 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f1cf8001690 con 0x7f1cfc006a70 2024-03-29T11:42:01.850 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.847+0000 7f1d037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:01.850 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.847+0000 7f1d037fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:01.850 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.847+0000 7f1d037fe640 10 monclient: dump: 2024-03-29T11:42:01.850 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:42:01.851 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:01.851 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:01.851 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:01.851 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:01.851 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:01.851 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:42:01.851 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:42:01.851 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:42:01.851 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:01.851 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.849+0000 7f1d09a94640 5 monclient: authenticate success, global_id 4484 2024-03-29T11:42:01.851 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.849+0000 7f1d09a94640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:01.852 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.849+0000 7f1d037fe640 1 -- 172.21.15.72:0/3728965688 <== mon.2 v1:172.21.15.72:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1cf8001860 con 0x7f1cfc006a70 2024-03-29T11:42:01.852 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.849+0000 7f1d037fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:01.853 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.849+0000 7f1d037fe640 1 -- 172.21.15.72:0/3728965688 <== mon.2 v1:172.21.15.72:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f1cf8001cb0 con 0x7f1cfc006a70 2024-03-29T11:42:01.853 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.849+0000 7f1d09a94640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:01.853 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.849+0000 7f1d09a94640 4 set_mon_vals no callback set 2024-03-29T11:42:01.853 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.849+0000 7f1d037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:01.853 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.849+0000 7f1d037fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:01.853 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.849+0000 7f1d037fe640 10 monclient: dump: 2024-03-29T11:42:01.853 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:42:01.853 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:01.853 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:01.853 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:01.853 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:01.853 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:01.853 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:42:01.853 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:42:01.853 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:42:01.853 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:01.854 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.850+0000 7f1d09a94640 10 monclient: shutdown 2024-03-29T11:42:01.854 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.850+0000 7f1d09a94640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:01.854 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.850+0000 7f1d09a94640 1 -- 172.21.15.72:0/3728965688 >> v1:172.21.15.72:6790/0 conn(0x7f1cfc006a70 legacy=0x7f1cfc0c7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:01.856 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.853+0000 7f1d09a94640 1 -- 172.21.15.72:0/3728965688 shutdown_connections 2024-03-29T11:42:01.857 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.853+0000 7f1d09a94640 1 -- 172.21.15.72:0/3728965688 >> 172.21.15.72:0/3728965688 conn(0x7f1cfc0058f0 msgr2=0x7f1cfc005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:01.857 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.854+0000 7f1d09a94640 1 -- 172.21.15.72:0/3728965688 shutdown_connections 2024-03-29T11:42:01.858 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.856+0000 7f1d09a94640 1 -- 172.21.15.72:0/3728965688 wait complete. 2024-03-29T11:42:01.859 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.857+0000 7f1d09a94640 10 monclient: build_initial_monmap 2024-03-29T11:42:01.859 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.857+0000 7f1d09a94640 10 monclient: monmap: 2024-03-29T11:42:01.860 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:42:01.860 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:01.860 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:42:01.860 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:42:01.860 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:01.860 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:01.860 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:42:01.860 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:42:01.860 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:42:01.860 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:01.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.857+0000 7f1d09a94640 1 librados: starting msgr at 2024-03-29T11:42:01.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.857+0000 7f1d09a94640 1 librados: starting objecter 2024-03-29T11:42:01.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.858+0000 7f1d09a94640 1 Processor -- start 2024-03-29T11:42:01.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.858+0000 7f1d09a94640 1 -- start start 2024-03-29T11:42:01.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.858+0000 7f1d09a94640 1 librados: setting wanted keys 2024-03-29T11:42:01.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.858+0000 7f1d09a94640 1 librados: calling monclient init 2024-03-29T11:42:01.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.858+0000 7f1d09a94640 10 monclient: init 2024-03-29T11:42:01.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.859+0000 7f1d09a94640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:01.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.859+0000 7f1d09a94640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:42:01.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.859+0000 7f1d09a94640 10 monclient(hunting): picked mon.noname-c con 0x7f1cfc006a70 addr v1:172.21.15.72:6790/0 2024-03-29T11:42:01.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.859+0000 7f1d08a92640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f1cfc006a70 0x7f1cfc158660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:51956/0 (socket says 172.21.15.72:51956) 2024-03-29T11:42:01.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.859+0000 7f1d08a92640 1 -- 172.21.15.72:0/3170229490 learned_addr learned my addr 172.21.15.72:0/3170229490 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:42:01.863 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.859+0000 7f1d017fa640 10 client.?.objecter ms_handle_connect 0x7f1cfc006a70 2024-03-29T11:42:01.863 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.860+0000 7f1d017fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:01.863 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.860+0000 7f1d09a94640 10 monclient(hunting): picked mon.noname-a con 0x7f1cfc0c1e70 addr v1:172.21.15.72:6789/0 2024-03-29T11:42:01.863 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.860+0000 7f1d09a94640 10 monclient(hunting): picked mon.noname-b con 0x7f1cfc158d70 addr v1:172.21.15.89:6789/0 2024-03-29T11:42:01.863 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.860+0000 7f1d09a94640 1 -- 172.21.15.72:0/3170229490 --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cfc02e4f0 con 0x7f1cfc0c1e70 2024-03-29T11:42:01.863 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.860+0000 7f1d09a94640 1 -- 172.21.15.72:0/3170229490 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cfc02e1c0 con 0x7f1cfc158d70 2024-03-29T11:42:01.863 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.860+0000 7f1d09a94640 1 -- 172.21.15.72:0/3170229490 --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cfc0c73e0 con 0x7f1cfc006a70 2024-03-29T11:42:01.864 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.860+0000 7f1d09a94640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:01.864 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.860+0000 7f1d09a94640 10 monclient(hunting): authenticate will time out at 1281.148838s 2024-03-29T11:42:01.866 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.863+0000 7f1d017fa640 10 client.?.objecter ms_handle_connect 0x7f1cfc0c1e70 2024-03-29T11:42:01.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.863+0000 7f1d017fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:01.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.864+0000 7f1d017fa640 1 -- 172.21.15.72:0/3170229490 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 82234018 0 0) 0x7f1cfc02e4f0 con 0x7f1cfc0c1e70 2024-03-29T11:42:01.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.864+0000 7f1d017fa640 10 monclient(hunting): _negotiate 2024-03-29T11:42:01.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.864+0000 7f1d017fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:01.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.864+0000 7f1d017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:01.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.864+0000 7f1d017fa640 10 monclient(hunting): my global_id is 4482 2024-03-29T11:42:01.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.864+0000 7f1d017fa640 1 -- 172.21.15.72:0/3170229490 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ce8004eb0 con 0x7f1cfc0c1e70 2024-03-29T11:42:01.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.864+0000 7f1d017fa640 1 -- 172.21.15.72:0/3170229490 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1219105641 0 0) 0x7f1cfc0c73e0 con 0x7f1cfc006a70 2024-03-29T11:42:01.870 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.864+0000 7f1d017fa640 10 monclient(hunting): _negotiate 2024-03-29T11:42:01.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.864+0000 7f1d017fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:01.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.864+0000 7f1d017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:01.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.864+0000 7f1d017fa640 10 monclient(hunting): my global_id is 4487 2024-03-29T11:42:01.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.864+0000 7f1d017fa640 1 -- 172.21.15.72:0/3170229490 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cfc02e4f0 con 0x7f1cfc006a70 2024-03-29T11:42:01.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.865+0000 7f1d017fa640 1 -- 172.21.15.72:0/3170229490 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1161186532 0 0) 0x7f1ce8004eb0 con 0x7f1cfc0c1e70 2024-03-29T11:42:01.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.865+0000 7f1d017fa640 1 -- 172.21.15.72:0/3170229490 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1cfc0c73e0 con 0x7f1cfc0c1e70 2024-03-29T11:42:01.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.865+0000 7f1d017fa640 1 -- 172.21.15.72:0/3170229490 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f1cf0001870 con 0x7f1cfc0c1e70 2024-03-29T11:42:01.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.865+0000 7f1d017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:01.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.865+0000 7f1d017fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:01.871 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.865+0000 7f1d017fa640 10 monclient(hunting): dump: 2024-03-29T11:42:01.871 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:42:01.872 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:01.872 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:01.872 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:01.872 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:01.872 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:01.872 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:42:01.872 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:42:01.872 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:42:01.872 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:01.872 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.865+0000 7f1d017fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:01.872 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.865+0000 7f1d017fa640 1 -- 172.21.15.72:0/3170229490 <== mon.0 v1:172.21.15.72:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4269922604 0 0) 0x7f1cfc0c73e0 con 0x7f1cfc0c1e70 2024-03-29T11:42:01.872 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.865+0000 7f1d017fa640 1 -- 172.21.15.72:0/3170229490 >> v1:172.21.15.72:6790/0 conn(0x7f1cfc006a70 legacy=0x7f1cfc158660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:01.872 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.865+0000 7f1d017fa640 1 -- 172.21.15.72:0/3170229490 >> v1:172.21.15.89:6789/0 conn(0x7f1cfc158d70 legacy=0x7f1cfc163ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:01.872 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.865+0000 7f1d017fa640 10 monclient: _finish_hunting 0 2024-03-29T11:42:01.872 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.865+0000 7f1d017fa640 1 monclient: found mon.a 2024-03-29T11:42:01.872 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.865+0000 7f1d017fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:01.873 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.866+0000 7f1d017fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:42:01.873 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.866+0000 7f1d017fa640 1 -- 172.21.15.72:0/3170229490 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cfc0c7570 con 0x7f1cfc0c1e70 2024-03-29T11:42:01.873 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.866+0000 7f1d017fa640 10 monclient: _finish_auth 0 2024-03-29T11:42:01.873 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.866+0000 7f1d017fa640 10 monclient: _check_auth_tickets 2024-03-29T11:42:01.873 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.866+0000 7f1d017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:01.873 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.866+0000 7f1d017fa640 1 -- 172.21.15.72:0/3170229490 <== mon.0 v1:172.21.15.72:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1cf00019d0 con 0x7f1cfc0c1e70 2024-03-29T11:42:01.873 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.866+0000 7f1d09a94640 5 monclient: authenticate success, global_id 4482 2024-03-29T11:42:01.873 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.866+0000 7f1d09a94640 10 monclient: _renew_subs 2024-03-29T11:42:01.873 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.866+0000 7f1d09a94640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:42:01.873 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.866+0000 7f1d09a94640 1 -- 172.21.15.72:0/3170229490 --> v1:172.21.15.72:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cfc031d50 con 0x7f1cfc0c1e70 2024-03-29T11:42:01.873 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.866+0000 7f1d09a94640 10 client.4482.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:01.873 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.866+0000 7f1d09a94640 10 monclient: _renew_subs 2024-03-29T11:42:01.873 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.866+0000 7f1d09a94640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:42:01.873 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.866+0000 7f1d09a94640 1 -- 172.21.15.72:0/3170229490 --> v1:172.21.15.72:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cfc02ebf0 con 0x7f1cfc0c1e70 2024-03-29T11:42:01.873 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.866+0000 7f1d09a94640 1 librados: init done 2024-03-29T11:42:01.874 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.866+0000 7f1d017fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:01.875 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.867+0000 7f1d02ffd640 4 set_mon_vals no callback set 2024-03-29T11:42:01.875 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.869+0000 7f1d017fa640 1 -- 172.21.15.72:0/3170229490 <== mon.0 v1:172.21.15.72:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f1cf00037d0 con 0x7f1cfc0c1e70 2024-03-29T11:42:01.875 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.869+0000 7f1d017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:01.875 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.869+0000 7f1d017fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:01.876 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.869+0000 7f1d017fa640 10 monclient: dump: 2024-03-29T11:42:01.876 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:42:01.876 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:01.876 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:01.876 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:01.876 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:01.876 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:01.876 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:42:01.876 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:42:01.876 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:42:01.876 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:01.876 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.869+0000 7f1d017fa640 1 -- 172.21.15.72:0/3170229490 <== mon.0 v1:172.21.15.72:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f1cf0019860 con 0x7f1cfc0c1e70 2024-03-29T11:42:01.876 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.870+0000 7f1d08a92640 10 monclient: get_auth_request con 0x7f1ce806db70 auth_method 0 2024-03-29T11:42:01.876 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.870+0000 7f1d017fa640 1 -- 172.21.15.72:0/3170229490 <== mon.0 v1:172.21.15.72:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f1cf0083870 con 0x7f1cfc0c1e70 2024-03-29T11:42:01.876 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.870+0000 7f1d017fa640 10 client.4482.objecter ms_dispatch 0x7f1d04000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:01.877 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.870+0000 7f1d017fa640 3 client.4482.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:01.877 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.870+0000 7f1d017fa640 3 client.4482.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:01.877 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.870+0000 7f1d017fa640 20 client.4482.objecter dump_active .. 0 homeless 2024-03-29T11:42:01.877 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.871+0000 7f1d017fa640 10 client.4482.objecter ms_handle_connect 0x7f1ce806db70 2024-03-29T11:42:01.877 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.871+0000 7f1d09a94640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:01.877 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.871+0000 7f1d09a94640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:01.877 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.871+0000 7f1d09a94640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:42:01.877 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.871+0000 7f1d09a94640 1 -- 172.21.15.72:0/3170229490 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ce4005080 con 0x7f1cfc0c1e70 2024-03-29T11:42:01.883 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.881+0000 7f1d017fa640 1 -- 172.21.15.72:0/3170229490 <== mon.0 v1:172.21.15.72:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1cf004f000 con 0x7f1cfc0c1e70 2024-03-29T11:42:01.884 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.881+0000 7f1d017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:01.884 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:01.881+0000 7f1d017fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:02.215 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:02.212+0000 7f1d09a94640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:02.215 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:02.212+0000 7f1d09a94640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:02.215 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:02.212+0000 7f1d09a94640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:42:02.215 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:02.212+0000 7f1d09a94640 1 -- 172.21.15.72:0/3170229490 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ce4005330 con 0x7f1cfc0c1e70 2024-03-29T11:42:02.216 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:02.214+0000 7f1d017fa640 1 -- 172.21.15.72:0/3170229490 <== mon.0 v1:172.21.15.72:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f1cf0051100 con 0x7f1cfc0c1e70 2024-03-29T11:42:02.216 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:02.214+0000 7f1d017fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:02.216 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:02.214+0000 7f1d017fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:02.219 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:42:02.219 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:02.219 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:02.217+0000 7f1d09a94640 10 librados: watch_flush enter 2024-03-29T11:42:02.220 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:02.217+0000 7f1d09a94640 10 librados: watch_flush exit 2024-03-29T11:42:02.220 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:02.217+0000 7f1d09a94640 20 client.4482.objecter shutdown clearing up homeless session... 2024-03-29T11:42:02.220 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:02.217+0000 7f1d09a94640 10 client.4482.objecter successfully canceled tick 2024-03-29T11:42:02.220 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:02.217+0000 7f1d09a94640 1 -- 172.21.15.72:0/3170229490 >> v1:172.21.15.72:6816/1578866102 conn(0x7f1ce806db70 legacy=0x7f1ce8070030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:02.221 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:02.218+0000 7f1d09a94640 10 monclient: shutdown 2024-03-29T11:42:02.221 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:02.218+0000 7f1d09a94640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:02.221 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:02.218+0000 7f1d09a94640 1 -- 172.21.15.72:0/3170229490 >> v1:172.21.15.72:6789/0 conn(0x7f1cfc0c1e70 legacy=0x7f1cfc031640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:02.221 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:02.218+0000 7f1d09a94640 1 -- 172.21.15.72:0/3170229490 shutdown_connections 2024-03-29T11:42:02.221 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:02.218+0000 7f1d09a94640 1 -- 172.21.15.72:0/3170229490 >> 172.21.15.72:0/3170229490 conn(0x7f1cfc0058f0 msgr2=0x7f1cfc033fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:02.221 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:02.218+0000 7f1d09a94640 1 -- 172.21.15.72:0/3170229490 shutdown_connections 2024-03-29T11:42:02.221 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:02.218+0000 7f1d09a94640 1 -- 172.21.15.72:0/3170229490 wait complete. 2024-03-29T11:42:02.221 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:02.219+0000 7f1d09a94640 1 librados: shutdown 2024-03-29T11:42:02.240 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:09.047 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:09.203 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.200+0000 7f55cc4d3640 10 monclient: get_monmap_and_config 2024-03-29T11:42:09.203 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.200+0000 7f55cc4d3640 10 monclient: build_initial_monmap 2024-03-29T11:42:09.203 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.200+0000 7f55cc4d3640 10 monclient: monmap: 2024-03-29T11:42:09.203 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:42:09.204 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:09.204 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:42:09.201286+0000 2024-03-29T11:42:09.204 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:42:09.201286+0000 2024-03-29T11:42:09.204 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:09.204 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:09.204 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:42:09.204 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:42:09.204 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:42:09.204 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:09.206 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.203+0000 7f55cc4d3640 1 Processor -- start 2024-03-29T11:42:09.206 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.204+0000 7f55cc4d3640 1 -- start start 2024-03-29T11:42:09.206 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.204+0000 7f55cc4d3640 10 monclient: init 2024-03-29T11:42:09.208 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.205+0000 7f55cc4d3640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:09.208 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.205+0000 7f55cc4d3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:42:09.209 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.206+0000 7f55cc4d3640 10 monclient(hunting): picked mon.noname-b con 0x7f55c40b0860 addr v1:172.21.15.72:6790/0 2024-03-29T11:42:09.209 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.206+0000 7f55cc4d3640 10 monclient(hunting): picked mon.noname-a con 0x7f55c413ef30 addr v1:172.21.15.72:6789/0 2024-03-29T11:42:09.209 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.206+0000 7f55cc4d3640 10 monclient(hunting): picked mon.noname-c con 0x7f55c406d9d0 addr v1:172.21.15.89:6789/0 2024-03-29T11:42:09.209 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.206+0000 7f55cc4d3640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c40b0c40 con 0x7f55c413ef30 2024-03-29T11:42:09.209 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.206+0000 7f55cc4d3640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c40aed10 con 0x7f55c406d9d0 2024-03-29T11:42:09.210 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.206+0000 7f55cc4d3640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c40ab1b0 con 0x7f55c40b0860 2024-03-29T11:42:09.210 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.206+0000 7f55cc4d3640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:09.210 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.206+0000 7f55cc4d3640 10 monclient(hunting): authenticate will time out at 1288.494380s 2024-03-29T11:42:09.215 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.209+0000 7f55ca26f640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f55c40b0860 0x7f55c413e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:39155/0 (socket says 172.21.15.72:39155) 2024-03-29T11:42:09.216 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.209+0000 7f55ca26f640 1 -- 172.21.15.72:0/1694057597 learned_addr learned my addr 172.21.15.72:0/1694057597 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:42:09.216 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.209+0000 7f55c926d640 1 -- 172.21.15.72:0/1694057597 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 566878982 0 0) 0x7f55c40ab1b0 con 0x7f55c40b0860 2024-03-29T11:42:09.216 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.209+0000 7f55c926d640 10 monclient(hunting): _negotiate 2024-03-29T11:42:09.216 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.209+0000 7f55c926d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:09.216 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.209+0000 7f55c926d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:09.216 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.209+0000 7f55c926d640 10 monclient(hunting): my global_id is 4490 2024-03-29T11:42:09.216 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.209+0000 7f55c926d640 1 -- 172.21.15.72:0/1694057597 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55b0003520 con 0x7f55c40b0860 2024-03-29T11:42:09.216 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.209+0000 7f55c926d640 1 -- 172.21.15.72:0/1694057597 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1945857005 0 0) 0x7f55b0003520 con 0x7f55c40b0860 2024-03-29T11:42:09.216 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.209+0000 7f55c926d640 1 -- 172.21.15.72:0/1694057597 >> v1:172.21.15.89:6789/0 conn(0x7f55c406d9d0 legacy=0x7f55c414a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:42:09.216 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.209+0000 7f55c926d640 1 -- 172.21.15.72:0/1694057597 >> v1:172.21.15.72:6789/0 conn(0x7f55c413ef30 legacy=0x7f55c40af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:42:09.216 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.210+0000 7f55c926d640 10 monclient: _finish_hunting 0 2024-03-29T11:42:09.216 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.210+0000 7f55c926d640 1 monclient: found mon.noname-b 2024-03-29T11:42:09.216 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.210+0000 7f55c926d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:09.216 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.210+0000 7f55c926d640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.72:6790/0 2024-03-29T11:42:09.217 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.210+0000 7f55c926d640 1 -- 172.21.15.72:0/1694057597 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c40ab490 con 0x7f55c40b0860 2024-03-29T11:42:09.217 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.210+0000 7f55c926d640 10 monclient: _finish_auth 0 2024-03-29T11:42:09.217 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.210+0000 7f55c926d640 10 monclient: _check_auth_tickets 2024-03-29T11:42:09.217 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.210+0000 7f55c926d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:09.217 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.210+0000 7f55c926d640 1 -- 172.21.15.72:0/1694057597 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f55c0001690 con 0x7f55c40b0860 2024-03-29T11:42:09.217 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.210+0000 7f55c926d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:09.217 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.210+0000 7f55c926d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:09.217 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.210+0000 7f55c926d640 10 monclient: dump: 2024-03-29T11:42:09.217 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:42:09.217 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:09.217 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:09.217 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:09.217 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:09.217 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:09.217 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:42:09.217 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:42:09.218 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:42:09.218 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:09.218 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.210+0000 7f55c926d640 1 -- 172.21.15.72:0/1694057597 <== mon.2 v1:172.21.15.72:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f55c0001860 con 0x7f55c40b0860 2024-03-29T11:42:09.218 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.210+0000 7f55c926d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:09.218 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.210+0000 7f55c926d640 1 -- 172.21.15.72:0/1694057597 <== mon.2 v1:172.21.15.72:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f55c0001cb0 con 0x7f55c40b0860 2024-03-29T11:42:09.218 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.210+0000 7f55c926d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:09.218 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.210+0000 7f55c926d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:09.218 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.210+0000 7f55c926d640 10 monclient: dump: 2024-03-29T11:42:09.218 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:42:09.218 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:09.218 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:09.218 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:09.218 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:09.218 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:09.219 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:42:09.219 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:42:09.219 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:42:09.219 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:09.219 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.211+0000 7f55cc4d3640 5 monclient: authenticate success, global_id 4490 2024-03-29T11:42:09.219 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.211+0000 7f55cc4d3640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:09.219 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.211+0000 7f55cc4d3640 4 set_mon_vals no callback set 2024-03-29T11:42:09.219 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.211+0000 7f55cc4d3640 10 monclient: shutdown 2024-03-29T11:42:09.219 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.212+0000 7f55cc4d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:09.219 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.212+0000 7f55cc4d3640 1 -- 172.21.15.72:0/1694057597 >> v1:172.21.15.72:6790/0 conn(0x7f55c40b0860 legacy=0x7f55c413e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:09.219 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.212+0000 7f55cc4d3640 1 -- 172.21.15.72:0/1694057597 shutdown_connections 2024-03-29T11:42:09.219 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.212+0000 7f55cc4d3640 1 -- 172.21.15.72:0/1694057597 >> 172.21.15.72:0/1694057597 conn(0x7f55c406b660 msgr2=0x7f55c406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:09.220 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.214+0000 7f55cc4d3640 1 -- 172.21.15.72:0/1694057597 shutdown_connections 2024-03-29T11:42:09.220 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.214+0000 7f55cc4d3640 1 -- 172.21.15.72:0/1694057597 wait complete. 2024-03-29T11:42:09.220 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.215+0000 7f55cc4d3640 10 monclient: build_initial_monmap 2024-03-29T11:42:09.220 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.215+0000 7f55cc4d3640 10 monclient: monmap: 2024-03-29T11:42:09.220 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:42:09.221 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:09.221 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:42:09.221 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:42:09.221 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:09.221 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:09.221 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:42:09.221 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:42:09.221 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:42:09.221 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:09.221 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.215+0000 7f55cc4d3640 1 librados: starting msgr at 2024-03-29T11:42:09.221 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.215+0000 7f55cc4d3640 1 librados: starting objecter 2024-03-29T11:42:09.221 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.215+0000 7f55cc4d3640 1 Processor -- start 2024-03-29T11:42:09.221 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.215+0000 7f55cc4d3640 1 -- start start 2024-03-29T11:42:09.221 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.215+0000 7f55cc4d3640 1 librados: setting wanted keys 2024-03-29T11:42:09.221 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.215+0000 7f55cc4d3640 1 librados: calling monclient init 2024-03-29T11:42:09.222 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.215+0000 7f55cc4d3640 10 monclient: init 2024-03-29T11:42:09.222 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.216+0000 7f55cc4d3640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:09.222 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.216+0000 7f55cc4d3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:42:09.225 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.219+0000 7f55cc4d3640 10 monclient(hunting): picked mon.noname-c con 0x7f55c406d9d0 addr v1:172.21.15.72:6790/0 2024-03-29T11:42:09.225 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.220+0000 7f55cc4d3640 10 monclient(hunting): picked mon.noname-b con 0x7f55c40b0860 addr v1:172.21.15.89:6789/0 2024-03-29T11:42:09.225 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.220+0000 7f55cc4d3640 10 monclient(hunting): picked mon.noname-a con 0x7f55c413ef30 addr v1:172.21.15.72:6789/0 2024-03-29T11:42:09.225 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.220+0000 7f55cc4d3640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c41dfdf0 con 0x7f55c413ef30 2024-03-29T11:42:09.225 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.220+0000 7f55cc4d3640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c40b24c0 con 0x7f55c40b0860 2024-03-29T11:42:09.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.220+0000 7f55cc4d3640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c4150920 con 0x7f55c406d9d0 2024-03-29T11:42:09.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.220+0000 7f55cc4d3640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:09.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.220+0000 7f55cc4d3640 10 monclient(hunting): authenticate will time out at 1288.508167s 2024-03-29T11:42:09.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.220+0000 7f55ca26f640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f55c406d9d0 0x7f55c4071140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:39219/0 (socket says 172.21.15.72:39219) 2024-03-29T11:42:09.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.220+0000 7f55ca26f640 1 -- 172.21.15.72:0/233842722 learned_addr learned my addr 172.21.15.72:0/233842722 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:42:09.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.220+0000 7f55baffd640 10 client.?.objecter ms_handle_connect 0x7f55c406d9d0 2024-03-29T11:42:09.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.220+0000 7f55baffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:09.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.220+0000 7f55baffd640 1 -- 172.21.15.72:0/233842722 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1239441738 0 0) 0x7f55c4150920 con 0x7f55c406d9d0 2024-03-29T11:42:09.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.220+0000 7f55baffd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:09.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.220+0000 7f55baffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:09.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.220+0000 7f55baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:09.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.220+0000 7f55baffd640 10 monclient(hunting): my global_id is 4493 2024-03-29T11:42:09.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.220+0000 7f55baffd640 1 -- 172.21.15.72:0/233842722 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55a8004eb0 con 0x7f55c406d9d0 2024-03-29T11:42:09.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.221+0000 7f55baffd640 1 -- 172.21.15.72:0/233842722 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1119043902 0 0) 0x7f55a8004eb0 con 0x7f55c406d9d0 2024-03-29T11:42:09.227 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.221+0000 7f55baffd640 1 -- 172.21.15.72:0/233842722 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55c4150920 con 0x7f55c406d9d0 2024-03-29T11:42:09.227 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.221+0000 7f55baffd640 1 -- 172.21.15.72:0/233842722 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f55c0001b70 con 0x7f55c406d9d0 2024-03-29T11:42:09.227 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.221+0000 7f55baffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:09.227 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.221+0000 7f55baffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:09.227 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.221+0000 7f55baffd640 10 monclient(hunting): dump: 2024-03-29T11:42:09.227 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:42:09.227 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:09.227 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:09.227 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:09.227 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:09.227 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:09.227 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:42:09.227 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:42:09.227 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:42:09.227 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:09.228 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.221+0000 7f55baffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:09.228 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.221+0000 7f55baffd640 1 -- 172.21.15.72:0/233842722 <== mon.2 v1:172.21.15.72:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1861686493 0 0) 0x7f55c4150920 con 0x7f55c406d9d0 2024-03-29T11:42:09.228 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.221+0000 7f55baffd640 1 -- 172.21.15.72:0/233842722 >> v1:172.21.15.89:6789/0 conn(0x7f55c40b0860 legacy=0x7f55c41defd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:42:09.228 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.223+0000 7f55baffd640 1 -- 172.21.15.72:0/233842722 >> v1:172.21.15.72:6789/0 conn(0x7f55c413ef30 legacy=0x7f55c41df6e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:42:09.230 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.224+0000 7f55baffd640 10 monclient: _finish_hunting 0 2024-03-29T11:42:09.230 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.224+0000 7f55baffd640 1 monclient: found mon.c 2024-03-29T11:42:09.230 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.224+0000 7f55baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:09.230 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.224+0000 7f55baffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:42:09.230 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.224+0000 7f55baffd640 1 -- 172.21.15.72:0/233842722 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c4070760 con 0x7f55c406d9d0 2024-03-29T11:42:09.230 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.224+0000 7f55baffd640 10 monclient: _finish_auth 0 2024-03-29T11:42:09.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.224+0000 7f55baffd640 10 monclient: _check_auth_tickets 2024-03-29T11:42:09.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.224+0000 7f55baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:09.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.224+0000 7f55cc4d3640 5 monclient: authenticate success, global_id 4493 2024-03-29T11:42:09.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.224+0000 7f55cc4d3640 10 monclient: _renew_subs 2024-03-29T11:42:09.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.224+0000 7f55cc4d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:42:09.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.224+0000 7f55cc4d3640 1 -- 172.21.15.72:0/233842722 --> v1:172.21.15.72:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55c4070a40 con 0x7f55c406d9d0 2024-03-29T11:42:09.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.224+0000 7f55cc4d3640 10 client.4493.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:09.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.224+0000 7f55cc4d3640 10 monclient: _renew_subs 2024-03-29T11:42:09.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.224+0000 7f55cc4d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:42:09.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.224+0000 7f55cc4d3640 1 -- 172.21.15.72:0/233842722 --> v1:172.21.15.72:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f55c406fb90 con 0x7f55c406d9d0 2024-03-29T11:42:09.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.224+0000 7f55cc4d3640 1 librados: init done 2024-03-29T11:42:09.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.226+0000 7f55b8ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:09.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.226+0000 7f55b8ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:09.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.226+0000 7f55b8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:42:09.232 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.226+0000 7f55b8ff9640 1 -- 172.21.15.72:0/233842722 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55ac005080 con 0x7f55c406d9d0 2024-03-29T11:42:09.232 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.228+0000 7f55baffd640 1 -- 172.21.15.72:0/233842722 <== mon.2 v1:172.21.15.72:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f55c0004bf0 con 0x7f55c406d9d0 2024-03-29T11:42:09.232 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.228+0000 7f55baffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:09.233 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.228+0000 7f55c8a6c640 4 set_mon_vals no callback set 2024-03-29T11:42:09.233 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.231+0000 7f55baffd640 1 -- 172.21.15.72:0/233842722 <== mon.2 v1:172.21.15.72:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f55c0005040 con 0x7f55c406d9d0 2024-03-29T11:42:09.234 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.231+0000 7f55baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:09.234 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.231+0000 7f55baffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:09.234 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.231+0000 7f55baffd640 10 monclient: dump: 2024-03-29T11:42:09.234 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:42:09.234 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:09.234 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:09.234 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:09.234 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:09.234 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:09.234 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:42:09.234 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:42:09.234 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:42:09.234 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:09.234 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.231+0000 7f55baffd640 1 -- 172.21.15.72:0/233842722 <== mon.2 v1:172.21.15.72:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f55c0005260 con 0x7f55c406d9d0 2024-03-29T11:42:09.235 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.231+0000 7f55baffd640 1 -- 172.21.15.72:0/233842722 <== mon.2 v1:172.21.15.72:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f55c0084d70 con 0x7f55c406d9d0 2024-03-29T11:42:09.235 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.231+0000 7f55baffd640 10 client.4493.objecter ms_dispatch 0x7f55c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:09.235 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.231+0000 7f55baffd640 3 client.4493.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:09.235 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.231+0000 7f55baffd640 3 client.4493.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:09.236 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.232+0000 7f55baffd640 20 client.4493.objecter dump_active .. 0 homeless 2024-03-29T11:42:09.236 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.233+0000 7f55c9a6e640 10 monclient: get_auth_request con 0x7f55a806dd50 auth_method 0 2024-03-29T11:42:09.237 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.235+0000 7f55baffd640 10 client.4493.objecter ms_handle_connect 0x7f55a806dd50 2024-03-29T11:42:09.243 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.240+0000 7f55baffd640 1 -- 172.21.15.72:0/233842722 <== mon.2 v1:172.21.15.72:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f55c00506d0 con 0x7f55c406d9d0 2024-03-29T11:42:09.243 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.240+0000 7f55baffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:09.243 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.240+0000 7f55baffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:09.577 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.574+0000 7f55b8ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:09.577 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.574+0000 7f55b8ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:09.578 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.574+0000 7f55b8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:42:09.578 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.574+0000 7f55b8ff9640 1 -- 172.21.15.72:0/233842722 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f55ac005330 con 0x7f55c406d9d0 2024-03-29T11:42:09.579 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.577+0000 7f55baffd640 1 -- 172.21.15.72:0/233842722 <== mon.2 v1:172.21.15.72:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f55c0052600 con 0x7f55c406d9d0 2024-03-29T11:42:09.579 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.577+0000 7f55baffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:09.580 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.577+0000 7f55baffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:09.582 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:42:09.582 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:09.586 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.580+0000 7f55b8ff9640 10 librados: watch_flush enter 2024-03-29T11:42:09.586 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.581+0000 7f55b8ff9640 10 librados: watch_flush exit 2024-03-29T11:42:09.586 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.581+0000 7f55b8ff9640 20 client.4493.objecter shutdown clearing up homeless session... 2024-03-29T11:42:09.586 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.581+0000 7f55b8ff9640 10 client.4493.objecter successfully canceled tick 2024-03-29T11:42:09.586 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.581+0000 7f55b8ff9640 1 -- 172.21.15.72:0/233842722 >> v1:172.21.15.72:6816/1578866102 conn(0x7f55a806dd50 legacy=0x7f55a8070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:09.587 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.581+0000 7f55b8ff9640 10 monclient: shutdown 2024-03-29T11:42:09.587 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.581+0000 7f55b8ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:09.587 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.581+0000 7f55b8ff9640 1 -- 172.21.15.72:0/233842722 >> v1:172.21.15.72:6790/0 conn(0x7f55c406d9d0 legacy=0x7f55c4071140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:09.587 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.581+0000 7f55b8ff9640 1 -- 172.21.15.72:0/233842722 shutdown_connections 2024-03-29T11:42:09.587 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.581+0000 7f55b8ff9640 1 -- 172.21.15.72:0/233842722 >> 172.21.15.72:0/233842722 conn(0x7f55c406b660 msgr2=0x7f55c413e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:09.587 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.582+0000 7f55b8ff9640 1 -- 172.21.15.72:0/233842722 shutdown_connections 2024-03-29T11:42:09.587 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.582+0000 7f55b8ff9640 1 -- 172.21.15.72:0/233842722 wait complete. 2024-03-29T11:42:09.587 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:09.582+0000 7f55b8ff9640 1 librados: shutdown 2024-03-29T11:42:09.605 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:16.412 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:16.576 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.572+0000 7f06011ac640 10 monclient: get_monmap_and_config 2024-03-29T11:42:16.576 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.572+0000 7f06011ac640 10 monclient: build_initial_monmap 2024-03-29T11:42:16.577 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.572+0000 7f06011ac640 10 monclient: monmap: 2024-03-29T11:42:16.577 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:42:16.577 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:16.577 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:42:16.573671+0000 2024-03-29T11:42:16.577 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:42:16.573671+0000 2024-03-29T11:42:16.577 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:16.577 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:16.577 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:42:16.577 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:42:16.577 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:42:16.577 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:16.578 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.575+0000 7f06011ac640 1 Processor -- start 2024-03-29T11:42:16.578 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.575+0000 7f06011ac640 1 -- start start 2024-03-29T11:42:16.578 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.575+0000 7f06011ac640 10 monclient: init 2024-03-29T11:42:16.580 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.576+0000 7f06011ac640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:16.580 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.576+0000 7f06011ac640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:42:16.580 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.576+0000 7f06011ac640 10 monclient(hunting): picked mon.noname-b con 0x7f05fc071e60 addr v1:172.21.15.72:6790/0 2024-03-29T11:42:16.580 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.576+0000 7f06011ac640 10 monclient(hunting): picked mon.noname-a con 0x7f05fc0a9c70 addr v1:172.21.15.72:6789/0 2024-03-29T11:42:16.580 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.576+0000 7f06011ac640 10 monclient(hunting): picked mon.noname-c con 0x7f05fc13f880 addr v1:172.21.15.89:6789/0 2024-03-29T11:42:16.581 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.576+0000 7f06011ac640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05fc072ce0 con 0x7f05fc0a9c70 2024-03-29T11:42:16.581 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.576+0000 7f06011ac640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05fc0aaeb0 con 0x7f05fc13f880 2024-03-29T11:42:16.581 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.576+0000 7f06011ac640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05fc145ac0 con 0x7f05fc071e60 2024-03-29T11:42:16.581 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.576+0000 7f06011ac640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:16.581 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.576+0000 7f06011ac640 10 monclient(hunting): authenticate will time out at 1295.864490s 2024-03-29T11:42:16.582 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.579+0000 7f05fb7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f05fc0a9c70 0x7f05fc1453b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49595/0 (socket says 172.21.15.72:49595) 2024-03-29T11:42:16.582 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.579+0000 7f05fb7fe640 1 -- 172.21.15.72:0/3348654786 learned_addr learned my addr 172.21.15.72:0/3348654786 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:42:16.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.581+0000 7f05faffd640 1 -- 172.21.15.72:0/3348654786 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4007261599 0 0) 0x7f05fc072ce0 con 0x7f05fc0a9c70 2024-03-29T11:42:16.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.581+0000 7f05faffd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:16.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.581+0000 7f05faffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:16.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.581+0000 7f05faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:16.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.581+0000 7f05faffd640 10 monclient(hunting): my global_id is 4485 2024-03-29T11:42:16.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.581+0000 7f05faffd640 1 -- 172.21.15.72:0/3348654786 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05e4003520 con 0x7f05fc0a9c70 2024-03-29T11:42:16.589 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.585+0000 7f05faffd640 1 -- 172.21.15.72:0/3348654786 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1746257591 0 0) 0x7f05e4003520 con 0x7f05fc0a9c70 2024-03-29T11:42:16.589 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.585+0000 7f05faffd640 1 -- 172.21.15.72:0/3348654786 >> v1:172.21.15.72:6790/0 conn(0x7f05fc071e60 legacy=0x7f05fc13f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:16.589 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.585+0000 7f05faffd640 1 -- 172.21.15.72:0/3348654786 >> v1:172.21.15.89:6789/0 conn(0x7f05fc13f880 legacy=0x7f05fc14d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:16.589 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.585+0000 7f05faffd640 10 monclient: _finish_hunting 0 2024-03-29T11:42:16.589 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.585+0000 7f05faffd640 1 monclient: found mon.noname-a 2024-03-29T11:42:16.589 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.585+0000 7f05faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:16.589 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.585+0000 7f05faffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.72:6789/0 2024-03-29T11:42:16.589 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.585+0000 7f05faffd640 1 -- 172.21.15.72:0/3348654786 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05fc145c50 con 0x7f05fc0a9c70 2024-03-29T11:42:16.589 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.585+0000 7f05faffd640 10 monclient: _finish_auth 0 2024-03-29T11:42:16.589 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.585+0000 7f05faffd640 10 monclient: _check_auth_tickets 2024-03-29T11:42:16.589 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.585+0000 7f05faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:16.589 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.585+0000 7f06011ac640 5 monclient: authenticate success, global_id 4485 2024-03-29T11:42:16.590 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.585+0000 7f06011ac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:16.590 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.585+0000 7f05faffd640 1 -- 172.21.15.72:0/3348654786 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f05f4001a10 con 0x7f05fc0a9c70 2024-03-29T11:42:16.590 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.585+0000 7f05faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:16.590 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.585+0000 7f05faffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:16.590 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.585+0000 7f05faffd640 10 monclient: dump: 2024-03-29T11:42:16.590 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:42:16.590 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:16.590 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:16.590 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:16.590 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:16.590 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:16.590 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:42:16.590 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:42:16.590 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:42:16.590 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:16.591 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.585+0000 7f06011ac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:16.591 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.586+0000 7f05faffd640 1 -- 172.21.15.72:0/3348654786 <== mon.0 v1:172.21.15.72:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f05f4001c30 con 0x7f05fc0a9c70 2024-03-29T11:42:16.591 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.586+0000 7f05faffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:16.592 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.586+0000 7f06011ac640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:16.592 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.586+0000 7f06011ac640 4 set_mon_vals no callback set 2024-03-29T11:42:16.592 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.586+0000 7f05faffd640 1 -- 172.21.15.72:0/3348654786 <== mon.0 v1:172.21.15.72:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f05f4001fc0 con 0x7f05fc0a9c70 2024-03-29T11:42:16.592 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.586+0000 7f05faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:16.592 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.586+0000 7f05faffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:16.592 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.586+0000 7f05faffd640 10 monclient: dump: 2024-03-29T11:42:16.592 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:42:16.592 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:16.592 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:16.592 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:16.592 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:16.592 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:16.593 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:42:16.593 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:42:16.593 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:42:16.593 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:16.593 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.587+0000 7f06011ac640 10 monclient: shutdown 2024-03-29T11:42:16.593 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.587+0000 7f06011ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:16.593 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.587+0000 7f06011ac640 1 -- 172.21.15.72:0/3348654786 >> v1:172.21.15.72:6789/0 conn(0x7f05fc0a9c70 legacy=0x7f05fc1453b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:16.593 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.587+0000 7f06011ac640 1 -- 172.21.15.72:0/3348654786 shutdown_connections 2024-03-29T11:42:16.593 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.587+0000 7f06011ac640 1 -- 172.21.15.72:0/3348654786 >> 172.21.15.72:0/3348654786 conn(0x7f05fc0b3a30 msgr2=0x7f05fc0b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:16.593 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.587+0000 7f06011ac640 1 -- 172.21.15.72:0/3348654786 shutdown_connections 2024-03-29T11:42:16.593 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.587+0000 7f06011ac640 1 -- 172.21.15.72:0/3348654786 wait complete. 2024-03-29T11:42:16.593 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.588+0000 7f06011ac640 10 monclient: build_initial_monmap 2024-03-29T11:42:16.593 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.588+0000 7f06011ac640 10 monclient: monmap: 2024-03-29T11:42:16.593 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:42:16.593 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:16.593 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:42:16.594 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:42:16.594 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:16.594 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:16.594 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:42:16.594 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:42:16.594 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:42:16.594 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:16.594 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.588+0000 7f06011ac640 1 librados: starting msgr at 2024-03-29T11:42:16.594 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.588+0000 7f06011ac640 1 librados: starting objecter 2024-03-29T11:42:16.594 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.589+0000 7f06011ac640 1 Processor -- start 2024-03-29T11:42:16.596 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.589+0000 7f06011ac640 1 -- start start 2024-03-29T11:42:16.596 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.589+0000 7f06011ac640 1 librados: setting wanted keys 2024-03-29T11:42:16.596 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.589+0000 7f06011ac640 1 librados: calling monclient init 2024-03-29T11:42:16.596 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.589+0000 7f06011ac640 10 monclient: init 2024-03-29T11:42:16.596 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.590+0000 7f06011ac640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:16.596 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.590+0000 7f06011ac640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:42:16.596 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.590+0000 7f06011ac640 10 monclient(hunting): picked mon.noname-c con 0x7f05fc071e60 addr v1:172.21.15.72:6790/0 2024-03-29T11:42:16.596 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.590+0000 7f06011ac640 10 monclient(hunting): picked mon.noname-a con 0x7f05fc0a9c70 addr v1:172.21.15.72:6789/0 2024-03-29T11:42:16.596 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.590+0000 7f06011ac640 10 monclient(hunting): picked mon.noname-b con 0x7f05fc13f880 addr v1:172.21.15.89:6789/0 2024-03-29T11:42:16.596 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.590+0000 7f06011ac640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05fc06f360 con 0x7f05fc0a9c70 2024-03-29T11:42:16.597 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.590+0000 7f06011ac640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05fc0704c0 con 0x7f05fc13f880 2024-03-29T11:42:16.597 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.590+0000 7f06011ac640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05fc1e3ef0 con 0x7f05fc071e60 2024-03-29T11:42:16.597 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.590+0000 7f06011ac640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:16.597 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.590+0000 7f06011ac640 10 monclient(hunting): authenticate will time out at 1295.878369s 2024-03-29T11:42:16.597 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.592+0000 7f05fb7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f05fc0a9c70 0x7f05fc06ec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49729/0 (socket says 172.21.15.72:49729) 2024-03-29T11:42:16.597 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.592+0000 7f05fb7fe640 1 -- 172.21.15.72:0/4213015259 learned_addr learned my addr 172.21.15.72:0/4213015259 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:42:16.597 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.593+0000 7f05f8ff9640 10 client.?.objecter ms_handle_connect 0x7f05fc0a9c70 2024-03-29T11:42:16.597 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.593+0000 7f05f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:16.597 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.593+0000 7f05f8ff9640 1 -- 172.21.15.72:0/4213015259 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3050358070 0 0) 0x7f05fc06f360 con 0x7f05fc0a9c70 2024-03-29T11:42:16.597 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.593+0000 7f05f8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:42:16.597 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.593+0000 7f05f8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:16.597 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.593+0000 7f05f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:16.597 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.593+0000 7f05f8ff9640 10 monclient(hunting): my global_id is 4488 2024-03-29T11:42:16.597 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.593+0000 7f05f8ff9640 1 -- 172.21.15.72:0/4213015259 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05e8004eb0 con 0x7f05fc0a9c70 2024-03-29T11:42:16.598 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.594+0000 7f05f8ff9640 1 -- 172.21.15.72:0/4213015259 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2793116364 0 0) 0x7f05e8004eb0 con 0x7f05fc0a9c70 2024-03-29T11:42:16.598 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.594+0000 7f05f8ff9640 1 -- 172.21.15.72:0/4213015259 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05fc06f360 con 0x7f05fc0a9c70 2024-03-29T11:42:16.598 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.594+0000 7f05f8ff9640 1 -- 172.21.15.72:0/4213015259 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f05f4001f40 con 0x7f05fc0a9c70 2024-03-29T11:42:16.598 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.594+0000 7f05f8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:16.598 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.594+0000 7f05f8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:16.598 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.594+0000 7f05f8ff9640 10 monclient(hunting): dump: 2024-03-29T11:42:16.599 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:42:16.599 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:16.599 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:16.599 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:16.599 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:16.599 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:16.599 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:42:16.599 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:42:16.599 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:42:16.599 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:16.599 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.594+0000 7f05f8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:16.600 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.597+0000 7f05f8ff9640 10 client.?.objecter ms_handle_connect 0x7f05fc13f880 2024-03-29T11:42:16.600 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.597+0000 7f05f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:16.603 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.598+0000 7f05f8ff9640 1 -- 172.21.15.72:0/4213015259 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3026488037 0 0) 0x7f05fc0704c0 con 0x7f05fc13f880 2024-03-29T11:42:16.603 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.598+0000 7f05f8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:42:16.603 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.598+0000 7f05f8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:16.603 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.598+0000 7f05f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:16.603 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.598+0000 7f05f8ff9640 10 monclient(hunting): my global_id is 4444 2024-03-29T11:42:16.603 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.598+0000 7f05f8ff9640 1 -- 172.21.15.72:0/4213015259 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05e8004eb0 con 0x7f05fc13f880 2024-03-29T11:42:16.603 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.598+0000 7f05f8ff9640 1 -- 172.21.15.72:0/4213015259 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2590110703 0 0) 0x7f05e8004eb0 con 0x7f05fc13f880 2024-03-29T11:42:16.603 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.598+0000 7f05f8ff9640 1 -- 172.21.15.72:0/4213015259 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05e80059e0 con 0x7f05fc13f880 2024-03-29T11:42:16.603 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.598+0000 7f05f8ff9640 1 -- 172.21.15.72:0/4213015259 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f05f00016d0 con 0x7f05fc13f880 2024-03-29T11:42:16.603 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.598+0000 7f05f8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:16.603 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.598+0000 7f05f8ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:16.604 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.598+0000 7f05f8ff9640 10 monclient(hunting): dump: 2024-03-29T11:42:16.604 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:42:16.604 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:16.604 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:16.604 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:16.604 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:16.604 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:16.604 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:42:16.604 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:42:16.604 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:42:16.604 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:16.604 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.599+0000 7f05f8ff9640 10 client.?.objecter ms_handle_connect 0x7f05fc071e60 2024-03-29T11:42:16.604 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.599+0000 7f05f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:16.604 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.599+0000 7f05f8ff9640 1 -- 172.21.15.72:0/4213015259 <== mon.1 v1:172.21.15.89:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2530167550 0 0) 0x7f05e80059e0 con 0x7f05fc13f880 2024-03-29T11:42:16.604 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.599+0000 7f05f8ff9640 1 -- 172.21.15.72:0/4213015259 >> v1:172.21.15.72:6790/0 conn(0x7f05fc071e60 legacy=0x7f05fc06de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:16.605 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.599+0000 7f05f8ff9640 1 -- 172.21.15.72:0/4213015259 >> v1:172.21.15.72:6789/0 conn(0x7f05fc0a9c70 legacy=0x7f05fc06ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:16.605 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.599+0000 7f05f8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:42:16.605 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.599+0000 7f05f8ff9640 1 monclient: found mon.b 2024-03-29T11:42:16.605 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.599+0000 7f05f8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:16.605 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.599+0000 7f05f8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:42:16.605 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.599+0000 7f05f8ff9640 1 -- 172.21.15.72:0/4213015259 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05fc1e1a40 con 0x7f05fc13f880 2024-03-29T11:42:16.605 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.599+0000 7f05f8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:42:16.605 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.599+0000 7f05f8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:42:16.605 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.599+0000 7f05f8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:16.605 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.599+0000 7f06011ac640 5 monclient: authenticate success, global_id 4444 2024-03-29T11:42:16.606 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.599+0000 7f06011ac640 10 monclient: _renew_subs 2024-03-29T11:42:16.606 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.599+0000 7f06011ac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:42:16.606 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.599+0000 7f06011ac640 1 -- 172.21.15.72:0/4213015259 --> v1:172.21.15.89:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05fc0708e0 con 0x7f05fc13f880 2024-03-29T11:42:16.606 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.599+0000 7f06011ac640 10 client.4444.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:16.606 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.599+0000 7f06011ac640 10 monclient: _renew_subs 2024-03-29T11:42:16.606 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.599+0000 7f06011ac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:42:16.606 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.599+0000 7f06011ac640 1 -- 172.21.15.72:0/4213015259 --> v1:172.21.15.89:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f05fc1e45f0 con 0x7f05fc13f880 2024-03-29T11:42:16.606 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.599+0000 7f06011ac640 1 librados: init done 2024-03-29T11:42:16.606 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.600+0000 7f05f8ff9640 1 -- 172.21.15.72:0/4213015259 <== mon.1 v1:172.21.15.89:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f05f0001bc0 con 0x7f05fc13f880 2024-03-29T11:42:16.606 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.600+0000 7f05f8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:16.606 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.600+0000 7f05fa7fc640 4 set_mon_vals no callback set 2024-03-29T11:42:16.606 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.600+0000 7f05f8ff9640 1 -- 172.21.15.72:0/4213015259 <== mon.1 v1:172.21.15.89:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f05f00016d0 con 0x7f05fc13f880 2024-03-29T11:42:16.606 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.600+0000 7f05f8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:16.606 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.600+0000 7f05f8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:16.606 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.600+0000 7f05f8ff9640 10 monclient: dump: 2024-03-29T11:42:16.607 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:42:16.607 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:16.607 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:16.607 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:16.607 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:16.607 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:16.607 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:42:16.607 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:42:16.607 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:42:16.607 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:16.607 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.602+0000 7f05f8ff9640 1 -- 172.21.15.72:0/4213015259 <== mon.1 v1:172.21.15.89:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f05f0019460 con 0x7f05fc13f880 2024-03-29T11:42:16.608 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.603+0000 7f06011ac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:16.608 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.603+0000 7f05fbfff640 10 monclient: get_auth_request con 0x7f05e806e140 auth_method 0 2024-03-29T11:42:16.608 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.603+0000 7f05f8ff9640 1 -- 172.21.15.72:0/4213015259 <== mon.1 v1:172.21.15.89:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f05f0053070 con 0x7f05fc13f880 2024-03-29T11:42:16.608 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.603+0000 7f05f8ff9640 10 client.4444.objecter ms_dispatch 0x7f05fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:16.608 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.603+0000 7f05f8ff9640 3 client.4444.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:16.608 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.603+0000 7f05f8ff9640 3 client.4444.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:16.608 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.603+0000 7f05f8ff9640 20 client.4444.objecter dump_active .. 0 homeless 2024-03-29T11:42:16.608 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.603+0000 7f06011ac640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:16.608 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.603+0000 7f06011ac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:42:16.608 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.603+0000 7f06011ac640 1 -- 172.21.15.72:0/4213015259 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05dc005080 con 0x7f05fc13f880 2024-03-29T11:42:16.608 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.603+0000 7f05f8ff9640 10 client.4444.objecter ms_handle_connect 0x7f05e806e140 2024-03-29T11:42:16.614 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.611+0000 7f05f8ff9640 1 -- 172.21.15.72:0/4213015259 <== mon.1 v1:172.21.15.89:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f05f004eb80 con 0x7f05fc13f880 2024-03-29T11:42:16.614 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.611+0000 7f05f8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:16.614 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.611+0000 7f05f8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:16.950 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.947+0000 7f06011ac640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:16.950 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.947+0000 7f06011ac640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:16.950 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.947+0000 7f06011ac640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:42:16.950 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.947+0000 7f06011ac640 1 -- 172.21.15.72:0/4213015259 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f05dc005330 con 0x7f05fc13f880 2024-03-29T11:42:16.950 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.948+0000 7f05f8ff9640 1 -- 172.21.15.72:0/4213015259 <== mon.1 v1:172.21.15.89:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f05f004eb80 con 0x7f05fc13f880 2024-03-29T11:42:16.951 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.948+0000 7f05f8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:16.951 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.948+0000 7f05f8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:16.953 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:42:16.953 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:16.957 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.953+0000 7f06011ac640 10 librados: watch_flush enter 2024-03-29T11:42:16.957 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.954+0000 7f06011ac640 10 librados: watch_flush exit 2024-03-29T11:42:16.957 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.954+0000 7f06011ac640 20 client.4444.objecter shutdown clearing up homeless session... 2024-03-29T11:42:16.957 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.954+0000 7f06011ac640 10 client.4444.objecter successfully canceled tick 2024-03-29T11:42:16.957 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.954+0000 7f06011ac640 1 -- 172.21.15.72:0/4213015259 >> v1:172.21.15.72:6816/1578866102 conn(0x7f05e806e140 legacy=0x7f05e8070600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:16.957 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.954+0000 7f06011ac640 10 monclient: shutdown 2024-03-29T11:42:16.957 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.954+0000 7f06011ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:16.958 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.954+0000 7f06011ac640 1 -- 172.21.15.72:0/4213015259 >> v1:172.21.15.89:6789/0 conn(0x7f05fc13f880 legacy=0x7f05fc1e1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:16.958 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.954+0000 7f06011ac640 1 -- 172.21.15.72:0/4213015259 shutdown_connections 2024-03-29T11:42:16.958 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.954+0000 7f06011ac640 1 -- 172.21.15.72:0/4213015259 >> 172.21.15.72:0/4213015259 conn(0x7f05fc0b3a30 msgr2=0x7f05fc0b02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:16.958 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.955+0000 7f06011ac640 1 -- 172.21.15.72:0/4213015259 shutdown_connections 2024-03-29T11:42:16.958 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.956+0000 7f06011ac640 1 -- 172.21.15.72:0/4213015259 wait complete. 2024-03-29T11:42:16.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:16.956+0000 7f06011ac640 1 librados: shutdown 2024-03-29T11:42:16.979 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:23.786 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:23.929 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.925+0000 7fcdf0086640 10 monclient: get_monmap_and_config 2024-03-29T11:42:23.929 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.925+0000 7fcdf0086640 10 monclient: build_initial_monmap 2024-03-29T11:42:23.929 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.925+0000 7fcdf0086640 10 monclient: monmap: 2024-03-29T11:42:23.930 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:42:23.930 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:23.930 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:42:23.926769+0000 2024-03-29T11:42:23.930 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:42:23.926769+0000 2024-03-29T11:42:23.930 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:23.930 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:23.930 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:42:23.930 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:42:23.930 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:42:23.930 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:23.931 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.926+0000 7fcdf0086640 1 Processor -- start 2024-03-29T11:42:23.931 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.928+0000 7fcdf0086640 1 -- start start 2024-03-29T11:42:23.931 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.928+0000 7fcdf0086640 10 monclient: init 2024-03-29T11:42:23.932 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.929+0000 7fcdf0086640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:23.932 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.929+0000 7fcdf0086640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:42:23.932 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.929+0000 7fcdf0086640 10 monclient(hunting): picked mon.noname-b con 0x7fcde80ae250 addr v1:172.21.15.72:6790/0 2024-03-29T11:42:23.933 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.929+0000 7fcdf0086640 10 monclient(hunting): picked mon.noname-a con 0x7fcde813f970 addr v1:172.21.15.72:6789/0 2024-03-29T11:42:23.933 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.929+0000 7fcdf0086640 10 monclient(hunting): picked mon.noname-c con 0x7fcde8142800 addr v1:172.21.15.89:6789/0 2024-03-29T11:42:23.933 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.929+0000 7fcdf0086640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcde80afe00 con 0x7fcde813f970 2024-03-29T11:42:23.933 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.929+0000 7fcdf0086640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcde8142c10 con 0x7fcde8142800 2024-03-29T11:42:23.933 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.929+0000 7fcdf0086640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcde80afb20 con 0x7fcde80ae250 2024-03-29T11:42:23.933 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.929+0000 7fcdf0086640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:23.933 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.929+0000 7fcdf0086640 10 monclient(hunting): authenticate will time out at 1303.217196s 2024-03-29T11:42:23.935 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.930+0000 7fcdede22640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7fcde80ae250 0x7fcde813f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:57940/0 (socket says 172.21.15.72:57940) 2024-03-29T11:42:23.936 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.930+0000 7fcdede22640 1 -- 172.21.15.72:0/754804990 learned_addr learned my addr 172.21.15.72:0/754804990 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:42:23.936 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.930+0000 7fcdece20640 1 -- 172.21.15.72:0/754804990 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3085733389 0 0) 0x7fcde80afb20 con 0x7fcde80ae250 2024-03-29T11:42:23.936 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.930+0000 7fcdece20640 10 monclient(hunting): _negotiate 2024-03-29T11:42:23.936 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.930+0000 7fcdece20640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:23.936 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.930+0000 7fcdece20640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:23.936 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.930+0000 7fcdece20640 10 monclient(hunting): my global_id is 4502 2024-03-29T11:42:23.936 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.930+0000 7fcdece20640 1 -- 172.21.15.72:0/754804990 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdd0003520 con 0x7fcde80ae250 2024-03-29T11:42:23.936 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.931+0000 7fcdece20640 1 -- 172.21.15.72:0/754804990 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2365081658 0 0) 0x7fcdd0003520 con 0x7fcde80ae250 2024-03-29T11:42:23.936 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.931+0000 7fcdece20640 1 -- 172.21.15.72:0/754804990 >> v1:172.21.15.89:6789/0 conn(0x7fcde8142800 legacy=0x7fcde814da10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:42:23.937 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.934+0000 7fcdece20640 1 -- 172.21.15.72:0/754804990 >> v1:172.21.15.72:6789/0 conn(0x7fcde813f970 legacy=0x7fcde81420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:42:23.937 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.934+0000 7fcdece20640 10 monclient: _finish_hunting 0 2024-03-29T11:42:23.937 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.934+0000 7fcdece20640 1 monclient: found mon.noname-b 2024-03-29T11:42:23.937 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.934+0000 7fcdece20640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:23.937 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.934+0000 7fcdece20640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.72:6790/0 2024-03-29T11:42:23.937 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.934+0000 7fcdece20640 1 -- 172.21.15.72:0/754804990 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde80aab80 con 0x7fcde80ae250 2024-03-29T11:42:23.938 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.934+0000 7fcdece20640 10 monclient: _finish_auth 0 2024-03-29T11:42:23.938 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.934+0000 7fcdece20640 10 monclient: _check_auth_tickets 2024-03-29T11:42:23.940 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.934+0000 7fcdece20640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:23.940 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.934+0000 7fcdece20640 1 -- 172.21.15.72:0/754804990 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fcddc001690 con 0x7fcde80ae250 2024-03-29T11:42:23.941 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.934+0000 7fcdece20640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:23.941 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.934+0000 7fcdece20640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:23.941 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.934+0000 7fcdece20640 10 monclient: dump: 2024-03-29T11:42:23.941 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:42:23.941 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:23.941 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:23.941 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:23.941 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:23.941 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:23.941 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:42:23.941 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:42:23.941 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:42:23.941 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:23.941 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.934+0000 7fcdf0086640 5 monclient: authenticate success, global_id 4502 2024-03-29T11:42:23.941 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.934+0000 7fcdf0086640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:23.942 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.934+0000 7fcdece20640 1 -- 172.21.15.72:0/754804990 <== mon.2 v1:172.21.15.72:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcddc001920 con 0x7fcde80ae250 2024-03-29T11:42:23.942 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.934+0000 7fcdece20640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:23.942 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.934+0000 7fcdece20640 1 -- 172.21.15.72:0/754804990 <== mon.2 v1:172.21.15.72:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fcddc001cb0 con 0x7fcde80ae250 2024-03-29T11:42:23.942 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.934+0000 7fcdece20640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:23.942 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.934+0000 7fcdece20640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:23.942 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.934+0000 7fcdece20640 10 monclient: dump: 2024-03-29T11:42:23.942 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:42:23.942 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:23.942 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:23.942 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:23.942 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:23.942 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:23.942 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:42:23.943 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:42:23.943 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:42:23.943 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:23.943 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.934+0000 7fcdf0086640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:23.943 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.934+0000 7fcdf0086640 4 set_mon_vals no callback set 2024-03-29T11:42:23.943 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.935+0000 7fcdf0086640 10 monclient: shutdown 2024-03-29T11:42:23.943 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.935+0000 7fcdf0086640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:23.943 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.935+0000 7fcdf0086640 1 -- 172.21.15.72:0/754804990 >> v1:172.21.15.72:6790/0 conn(0x7fcde80ae250 legacy=0x7fcde813f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:23.943 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.935+0000 7fcdf0086640 1 -- 172.21.15.72:0/754804990 shutdown_connections 2024-03-29T11:42:23.943 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.935+0000 7fcdf0086640 1 -- 172.21.15.72:0/754804990 >> 172.21.15.72:0/754804990 conn(0x7fcde80b35e0 msgr2=0x7fcde80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:23.943 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.936+0000 7fcdf0086640 1 -- 172.21.15.72:0/754804990 shutdown_connections 2024-03-29T11:42:23.943 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.936+0000 7fcdf0086640 1 -- 172.21.15.72:0/754804990 wait complete. 2024-03-29T11:42:23.943 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.937+0000 7fcdf0086640 10 monclient: build_initial_monmap 2024-03-29T11:42:23.943 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.937+0000 7fcdf0086640 10 monclient: monmap: 2024-03-29T11:42:23.943 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:42:23.944 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:23.944 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:42:23.944 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:42:23.944 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:23.944 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:23.944 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:42:23.944 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:42:23.945 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:42:23.945 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:23.945 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.937+0000 7fcdf0086640 1 librados: starting msgr at 2024-03-29T11:42:23.945 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.937+0000 7fcdf0086640 1 librados: starting objecter 2024-03-29T11:42:23.945 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.938+0000 7fcdf0086640 1 Processor -- start 2024-03-29T11:42:23.951 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.938+0000 7fcdf0086640 1 -- start start 2024-03-29T11:42:23.951 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.938+0000 7fcdf0086640 1 librados: setting wanted keys 2024-03-29T11:42:23.951 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.938+0000 7fcdf0086640 1 librados: calling monclient init 2024-03-29T11:42:23.951 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.938+0000 7fcdf0086640 10 monclient: init 2024-03-29T11:42:23.951 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.938+0000 7fcdf0086640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:23.951 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.938+0000 7fcdf0086640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:42:23.951 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.938+0000 7fcdf0086640 10 monclient(hunting): picked mon.noname-b con 0x7fcde80ae250 addr v1:172.21.15.89:6789/0 2024-03-29T11:42:23.952 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.938+0000 7fcdf0086640 10 monclient(hunting): picked mon.noname-a con 0x7fcde8142800 addr v1:172.21.15.72:6789/0 2024-03-29T11:42:23.952 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.938+0000 7fcdf0086640 10 monclient(hunting): picked mon.noname-c con 0x7fcde8144760 addr v1:172.21.15.72:6790/0 2024-03-29T11:42:23.952 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.938+0000 7fcdf0086640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcde8146310 con 0x7fcde8142800 2024-03-29T11:42:23.952 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.938+0000 7fcdf0086640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcde8145f30 con 0x7fcde80ae250 2024-03-29T11:42:23.952 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.938+0000 7fcdf0086640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcde81434b0 con 0x7fcde8144760 2024-03-29T11:42:23.952 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.939+0000 7fcdf0086640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:23.952 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.939+0000 7fcdf0086640 10 monclient(hunting): authenticate will time out at 1303.227013s 2024-03-29T11:42:23.952 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.941+0000 7fcdee623640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7fcde8144760 0x7fcde81e7240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:59222/0 (socket says 172.21.15.72:59222) 2024-03-29T11:42:23.952 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.941+0000 7fcdee623640 1 -- 172.21.15.72:0/2999074453 learned_addr learned my addr 172.21.15.72:0/2999074453 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:42:23.952 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.941+0000 7fcde67fc640 10 client.?.objecter ms_handle_connect 0x7fcde8144760 2024-03-29T11:42:23.952 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.942+0000 7fcde67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:23.952 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.942+0000 7fcde67fc640 1 -- 172.21.15.72:0/2999074453 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3136305578 0 0) 0x7fcde81434b0 con 0x7fcde8144760 2024-03-29T11:42:23.952 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.942+0000 7fcde67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:23.952 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.942+0000 7fcde67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:23.953 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.942+0000 7fcde67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:23.953 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.942+0000 7fcde67fc640 10 monclient(hunting): my global_id is 4505 2024-03-29T11:42:23.953 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.942+0000 7fcde67fc640 1 -- 172.21.15.72:0/2999074453 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdd4004eb0 con 0x7fcde8144760 2024-03-29T11:42:23.953 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.942+0000 7fcde67fc640 1 -- 172.21.15.72:0/2999074453 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2163682272 0 0) 0x7fcdd4004eb0 con 0x7fcde8144760 2024-03-29T11:42:23.953 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.943+0000 7fcde67fc640 1 -- 172.21.15.72:0/2999074453 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcde81434b0 con 0x7fcde8144760 2024-03-29T11:42:23.953 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.943+0000 7fcde67fc640 1 -- 172.21.15.72:0/2999074453 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fcde0001ad0 con 0x7fcde8144760 2024-03-29T11:42:23.953 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.943+0000 7fcde67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:23.953 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.943+0000 7fcde67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:23.953 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.943+0000 7fcde67fc640 10 monclient(hunting): dump: 2024-03-29T11:42:23.953 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:42:23.953 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:23.953 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:23.953 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:23.953 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:23.954 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:23.954 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:42:23.954 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:42:23.954 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:42:23.954 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:23.954 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.943+0000 7fcde67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:23.954 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.943+0000 7fcde67fc640 1 -- 172.21.15.72:0/2999074453 <== mon.2 v1:172.21.15.72:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3352524804 0 0) 0x7fcde81434b0 con 0x7fcde8144760 2024-03-29T11:42:23.954 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.943+0000 7fcde67fc640 1 -- 172.21.15.72:0/2999074453 >> v1:172.21.15.89:6789/0 conn(0x7fcde80ae250 legacy=0x7fcde8144050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:23.954 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.943+0000 7fcde67fc640 1 -- 172.21.15.72:0/2999074453 >> v1:172.21.15.72:6789/0 conn(0x7fcde8142800 legacy=0x7fcde81e0e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:42:23.954 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.944+0000 7fcde67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:42:23.955 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.944+0000 7fcde67fc640 1 monclient: found mon.c 2024-03-29T11:42:23.955 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.945+0000 7fcde67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:23.955 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.945+0000 7fcde67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:42:23.955 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.945+0000 7fcde67fc640 1 -- 172.21.15.72:0/2999074453 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde8143640 con 0x7fcde8144760 2024-03-29T11:42:23.955 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.945+0000 7fcde67fc640 10 monclient: _finish_auth 0 2024-03-29T11:42:23.955 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.945+0000 7fcde67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:42:23.955 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.945+0000 7fcde67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:23.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.945+0000 7fcdf0086640 5 monclient: authenticate success, global_id 4505 2024-03-29T11:42:23.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.945+0000 7fcdf0086640 10 monclient: _renew_subs 2024-03-29T11:42:23.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.945+0000 7fcdf0086640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:42:23.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.945+0000 7fcdf0086640 1 -- 172.21.15.72:0/2999074453 --> v1:172.21.15.72:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcde81e1540 con 0x7fcde8144760 2024-03-29T11:42:23.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.945+0000 7fcdf0086640 10 client.4505.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:23.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.945+0000 7fcdf0086640 10 monclient: _renew_subs 2024-03-29T11:42:23.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.945+0000 7fcdf0086640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:42:23.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.945+0000 7fcdf0086640 1 -- 172.21.15.72:0/2999074453 --> v1:172.21.15.72:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcde81e7950 con 0x7fcde8144760 2024-03-29T11:42:23.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.945+0000 7fcdf0086640 1 librados: init done 2024-03-29T11:42:23.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.946+0000 7fcde67fc640 1 -- 172.21.15.72:0/2999074453 <== mon.2 v1:172.21.15.72:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcde0001c30 con 0x7fcde8144760 2024-03-29T11:42:23.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.946+0000 7fcde67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:23.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.946+0000 7fcde67fc640 1 -- 172.21.15.72:0/2999074453 <== mon.2 v1:172.21.15.72:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fcde0004d80 con 0x7fcde8144760 2024-03-29T11:42:23.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.946+0000 7fcde7fff640 4 set_mon_vals no callback set 2024-03-29T11:42:23.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.946+0000 7fcde67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:23.956 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.946+0000 7fcde67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:23.957 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.946+0000 7fcde67fc640 10 monclient: dump: 2024-03-29T11:42:23.957 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:42:23.957 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:23.957 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:23.957 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:23.957 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:23.957 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:23.957 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:42:23.957 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:42:23.957 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:42:23.957 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:23.957 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.946+0000 7fcdf0086640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:23.957 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.947+0000 7fcdf0086640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:23.957 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.947+0000 7fcdf0086640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:42:23.957 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.947+0000 7fcdf0086640 1 -- 172.21.15.72:0/2999074453 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdc8005080 con 0x7fcde8144760 2024-03-29T11:42:23.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.948+0000 7fcde67fc640 1 -- 172.21.15.72:0/2999074453 <== mon.2 v1:172.21.15.72:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7fcde001ad50 con 0x7fcde8144760 2024-03-29T11:42:23.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.949+0000 7fcde67fc640 1 -- 172.21.15.72:0/2999074453 <== mon.2 v1:172.21.15.72:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7fcde0084da0 con 0x7fcde8144760 2024-03-29T11:42:23.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.949+0000 7fcde67fc640 10 client.4505.objecter ms_dispatch 0x7fcde8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:23.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.949+0000 7fcdede22640 10 monclient: get_auth_request con 0x7fcdd406dc20 auth_method 0 2024-03-29T11:42:23.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.950+0000 7fcde67fc640 3 client.4505.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:23.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.950+0000 7fcde67fc640 3 client.4505.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:23.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.950+0000 7fcde67fc640 20 client.4505.objecter dump_active .. 0 homeless 2024-03-29T11:42:23.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.950+0000 7fcde67fc640 10 client.4505.objecter ms_handle_connect 0x7fcdd406dc20 2024-03-29T11:42:23.960 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.956+0000 7fcde67fc640 1 -- 172.21.15.72:0/2999074453 <== mon.2 v1:172.21.15.72:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcde0050680 con 0x7fcde8144760 2024-03-29T11:42:23.960 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.957+0000 7fcde67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:23.960 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:23.957+0000 7fcde67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:24.284 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:24.281+0000 7fcdf0086640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:24.284 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:24.281+0000 7fcdf0086640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:24.284 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:24.281+0000 7fcdf0086640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:42:24.284 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:24.281+0000 7fcdf0086640 1 -- 172.21.15.72:0/2999074453 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcdc8005330 con 0x7fcde8144760 2024-03-29T11:42:24.286 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:42:24.286 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:24.286 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:24.282+0000 7fcde67fc640 1 -- 172.21.15.72:0/2999074453 <== mon.2 v1:172.21.15.72:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7fcde00525b0 con 0x7fcde8144760 2024-03-29T11:42:24.286 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:24.282+0000 7fcde67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:24.286 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:24.282+0000 7fcde67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:24.289 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:24.286+0000 7fcdf0086640 10 librados: watch_flush enter 2024-03-29T11:42:24.289 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:24.286+0000 7fcdf0086640 10 librados: watch_flush exit 2024-03-29T11:42:24.290 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:24.286+0000 7fcdf0086640 20 client.4505.objecter shutdown clearing up homeless session... 2024-03-29T11:42:24.290 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:24.286+0000 7fcdf0086640 10 client.4505.objecter successfully canceled tick 2024-03-29T11:42:24.290 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:24.286+0000 7fcdf0086640 1 -- 172.21.15.72:0/2999074453 >> v1:172.21.15.72:6816/1578866102 conn(0x7fcdd406dc20 legacy=0x7fcdd40700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:24.290 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:24.286+0000 7fcdf0086640 10 monclient: shutdown 2024-03-29T11:42:24.290 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:24.286+0000 7fcdf0086640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:24.290 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:24.286+0000 7fcdf0086640 1 -- 172.21.15.72:0/2999074453 >> v1:172.21.15.72:6790/0 conn(0x7fcde8144760 legacy=0x7fcde81e7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:24.290 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:24.286+0000 7fcdf0086640 1 -- 172.21.15.72:0/2999074453 shutdown_connections 2024-03-29T11:42:24.290 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:24.286+0000 7fcdf0086640 1 -- 172.21.15.72:0/2999074453 >> 172.21.15.72:0/2999074453 conn(0x7fcde80b35e0 msgr2=0x7fcde80b0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:24.290 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:24.286+0000 7fcdf0086640 1 -- 172.21.15.72:0/2999074453 shutdown_connections 2024-03-29T11:42:24.290 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:24.286+0000 7fcdf0086640 1 -- 172.21.15.72:0/2999074453 wait complete. 2024-03-29T11:42:24.290 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:24.287+0000 7fcdf0086640 1 librados: shutdown 2024-03-29T11:42:24.308 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:31.115 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:31.263 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.259+0000 7ff90fd1c640 10 monclient: get_monmap_and_config 2024-03-29T11:42:31.263 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.259+0000 7ff90fd1c640 10 monclient: build_initial_monmap 2024-03-29T11:42:31.263 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.260+0000 7ff90fd1c640 10 monclient: monmap: 2024-03-29T11:42:31.263 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:42:31.263 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:31.263 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:42:31.260942+0000 2024-03-29T11:42:31.263 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:42:31.260942+0000 2024-03-29T11:42:31.263 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:31.263 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:31.264 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:42:31.264 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:42:31.264 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:42:31.264 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:31.264 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.261+0000 7ff90fd1c640 1 Processor -- start 2024-03-29T11:42:31.264 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.261+0000 7ff90fd1c640 1 -- start start 2024-03-29T11:42:31.264 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.261+0000 7ff90fd1c640 10 monclient: init 2024-03-29T11:42:31.266 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.262+0000 7ff90fd1c640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:31.266 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.262+0000 7ff90fd1c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:42:31.266 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.262+0000 7ff90fd1c640 10 monclient(hunting): picked mon.noname-a con 0x7ff9080708a0 addr v1:172.21.15.72:6789/0 2024-03-29T11:42:31.266 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.262+0000 7ff90fd1c640 10 monclient(hunting): picked mon.noname-c con 0x7ff90806fe90 addr v1:172.21.15.89:6789/0 2024-03-29T11:42:31.266 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.262+0000 7ff90fd1c640 10 monclient(hunting): picked mon.noname-b con 0x7ff908143c50 addr v1:172.21.15.72:6790/0 2024-03-29T11:42:31.266 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.262+0000 7ff90fd1c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff908072630 con 0x7ff9080708a0 2024-03-29T11:42:31.266 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.262+0000 7ff90fd1c640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff908071410 con 0x7ff90806fe90 2024-03-29T11:42:31.266 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.262+0000 7ff90fd1c640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9080afaa0 con 0x7ff908143c50 2024-03-29T11:42:31.266 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.262+0000 7ff90fd1c640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:31.267 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.262+0000 7ff90fd1c640 10 monclient(hunting): authenticate will time out at 1310.550356s 2024-03-29T11:42:31.269 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.266+0000 7ff90dab8640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7ff9080708a0 0x7ff9080af390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:53042/0 (socket says 172.21.15.72:53042) 2024-03-29T11:42:31.269 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.266+0000 7ff90dab8640 1 -- 172.21.15.72:0/2802664900 learned_addr learned my addr 172.21.15.72:0/2802664900 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:42:31.270 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.267+0000 7ff90cab6640 1 -- 172.21.15.72:0/2802664900 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 453571925 0 0) 0x7ff908072630 con 0x7ff9080708a0 2024-03-29T11:42:31.270 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.267+0000 7ff90cab6640 10 monclient(hunting): _negotiate 2024-03-29T11:42:31.270 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.267+0000 7ff90cab6640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:31.270 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.267+0000 7ff90cab6640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:31.270 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.267+0000 7ff90cab6640 10 monclient(hunting): my global_id is 4491 2024-03-29T11:42:31.270 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.267+0000 7ff90cab6640 1 -- 172.21.15.72:0/2802664900 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8f4003520 con 0x7ff9080708a0 2024-03-29T11:42:31.272 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.267+0000 7ff90cab6640 1 -- 172.21.15.72:0/2802664900 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 473139777 0 0) 0x7ff8f4003520 con 0x7ff9080708a0 2024-03-29T11:42:31.272 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.267+0000 7ff90cab6640 1 -- 172.21.15.72:0/2802664900 >> v1:172.21.15.72:6790/0 conn(0x7ff908143c50 legacy=0x7ff90814b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:31.272 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.267+0000 7ff90cab6640 1 -- 172.21.15.72:0/2802664900 >> v1:172.21.15.89:6789/0 conn(0x7ff90806fe90 legacy=0x7ff908143540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:42:31.272 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.267+0000 7ff90cab6640 10 monclient: _finish_hunting 0 2024-03-29T11:42:31.272 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.267+0000 7ff90cab6640 1 monclient: found mon.noname-a 2024-03-29T11:42:31.272 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.267+0000 7ff90cab6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:31.272 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.267+0000 7ff90cab6640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.72:6789/0 2024-03-29T11:42:31.272 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.267+0000 7ff90cab6640 1 -- 172.21.15.72:0/2802664900 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9080ac8b0 con 0x7ff9080708a0 2024-03-29T11:42:31.273 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.267+0000 7ff90cab6640 10 monclient: _finish_auth 0 2024-03-29T11:42:31.273 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.267+0000 7ff90cab6640 10 monclient: _check_auth_tickets 2024-03-29T11:42:31.273 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.267+0000 7ff90cab6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:31.273 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.267+0000 7ff90cab6640 1 -- 172.21.15.72:0/2802664900 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7ff904001690 con 0x7ff9080708a0 2024-03-29T11:42:31.273 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.267+0000 7ff90cab6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:31.273 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.267+0000 7ff90cab6640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:31.273 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.267+0000 7ff90cab6640 10 monclient: dump: 2024-03-29T11:42:31.273 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:42:31.273 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:31.273 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:31.273 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:31.273 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:31.273 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:31.273 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:42:31.273 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:42:31.274 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:42:31.274 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:31.274 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.267+0000 7ff90fd1c640 5 monclient: authenticate success, global_id 4491 2024-03-29T11:42:31.274 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.267+0000 7ff90fd1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:31.274 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.267+0000 7ff90cab6640 1 -- 172.21.15.72:0/2802664900 <== mon.0 v1:172.21.15.72:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff904001860 con 0x7ff9080708a0 2024-03-29T11:42:31.274 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.267+0000 7ff90cab6640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:31.274 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.267+0000 7ff90cab6640 1 -- 172.21.15.72:0/2802664900 <== mon.0 v1:172.21.15.72:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7ff904001c70 con 0x7ff9080708a0 2024-03-29T11:42:31.274 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.267+0000 7ff90cab6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:31.274 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.267+0000 7ff90cab6640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:31.274 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.267+0000 7ff90cab6640 10 monclient: dump: 2024-03-29T11:42:31.274 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:42:31.274 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:31.274 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:31.274 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:31.274 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:31.275 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:31.275 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:42:31.275 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:42:31.275 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:42:31.275 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:31.275 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.267+0000 7ff90fd1c640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:31.275 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.267+0000 7ff90fd1c640 4 set_mon_vals no callback set 2024-03-29T11:42:31.275 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.268+0000 7ff90fd1c640 10 monclient: shutdown 2024-03-29T11:42:31.275 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.268+0000 7ff90fd1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:31.275 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.268+0000 7ff90fd1c640 1 -- 172.21.15.72:0/2802664900 >> v1:172.21.15.72:6789/0 conn(0x7ff9080708a0 legacy=0x7ff9080af390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:31.275 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.269+0000 7ff90fd1c640 1 -- 172.21.15.72:0/2802664900 shutdown_connections 2024-03-29T11:42:31.275 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.269+0000 7ff90fd1c640 1 -- 172.21.15.72:0/2802664900 >> 172.21.15.72:0/2802664900 conn(0x7ff9080675b0 msgr2=0x7ff9080b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:31.275 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.269+0000 7ff90fd1c640 1 -- 172.21.15.72:0/2802664900 shutdown_connections 2024-03-29T11:42:31.275 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.269+0000 7ff90fd1c640 1 -- 172.21.15.72:0/2802664900 wait complete. 2024-03-29T11:42:31.277 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.270+0000 7ff90fd1c640 10 monclient: build_initial_monmap 2024-03-29T11:42:31.277 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.270+0000 7ff90fd1c640 10 monclient: monmap: 2024-03-29T11:42:31.277 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:42:31.277 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:31.277 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:42:31.277 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:42:31.277 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:31.277 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:31.278 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:42:31.278 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:42:31.278 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:42:31.278 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:31.278 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.272+0000 7ff90fd1c640 1 librados: starting msgr at 2024-03-29T11:42:31.278 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.272+0000 7ff90fd1c640 1 librados: starting objecter 2024-03-29T11:42:31.279 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.272+0000 7ff90fd1c640 1 Processor -- start 2024-03-29T11:42:31.279 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.272+0000 7ff90fd1c640 1 -- start start 2024-03-29T11:42:31.279 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.272+0000 7ff90fd1c640 1 librados: setting wanted keys 2024-03-29T11:42:31.279 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.272+0000 7ff90fd1c640 1 librados: calling monclient init 2024-03-29T11:42:31.279 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.272+0000 7ff90fd1c640 10 monclient: init 2024-03-29T11:42:31.279 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.273+0000 7ff90fd1c640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:31.279 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.273+0000 7ff90fd1c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:42:31.279 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.273+0000 7ff90fd1c640 10 monclient(hunting): picked mon.noname-c con 0x7ff90806fe90 addr v1:172.21.15.72:6790/0 2024-03-29T11:42:31.279 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.273+0000 7ff90fd1c640 10 monclient(hunting): picked mon.noname-a con 0x7ff9080708a0 addr v1:172.21.15.72:6789/0 2024-03-29T11:42:31.281 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.274+0000 7ff90fd1c640 10 monclient(hunting): picked mon.noname-b con 0x7ff908143c50 addr v1:172.21.15.89:6789/0 2024-03-29T11:42:31.281 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.276+0000 7ff90fd1c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff90813f4d0 con 0x7ff9080708a0 2024-03-29T11:42:31.281 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.276+0000 7ff90fd1c640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff908133c30 con 0x7ff908143c50 2024-03-29T11:42:31.281 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.276+0000 7ff90fd1c640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9080ab1f0 con 0x7ff90806fe90 2024-03-29T11:42:31.282 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.276+0000 7ff90fd1c640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:31.282 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.276+0000 7ff90fd1c640 10 monclient(hunting): authenticate will time out at 1310.564173s 2024-03-29T11:42:31.282 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.277+0000 7ff90dab8640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7ff90806fe90 0x7ff9080aa4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:56793/0 (socket says 172.21.15.72:56793) 2024-03-29T11:42:31.282 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.277+0000 7ff90dab8640 1 -- 172.21.15.72:0/2653384946 learned_addr learned my addr 172.21.15.72:0/2653384946 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:42:31.282 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.277+0000 7ff8fe7fc640 10 client.?.objecter ms_handle_connect 0x7ff90806fe90 2024-03-29T11:42:31.282 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.277+0000 7ff8fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:31.282 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.278+0000 7ff8fe7fc640 10 client.?.objecter ms_handle_connect 0x7ff9080708a0 2024-03-29T11:42:31.282 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.278+0000 7ff8fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:31.282 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.278+0000 7ff8fe7fc640 1 -- 172.21.15.72:0/2653384946 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1233875974 0 0) 0x7ff9080ab1f0 con 0x7ff90806fe90 2024-03-29T11:42:31.282 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.278+0000 7ff8fe7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:31.282 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.278+0000 7ff8fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:31.282 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.278+0000 7ff8fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:31.282 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.278+0000 7ff8fe7fc640 10 monclient(hunting): my global_id is 4508 2024-03-29T11:42:31.282 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.278+0000 7ff8fe7fc640 1 -- 172.21.15.72:0/2653384946 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8ec004eb0 con 0x7ff90806fe90 2024-03-29T11:42:31.285 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.280+0000 7ff8fe7fc640 10 client.?.objecter ms_handle_connect 0x7ff908143c50 2024-03-29T11:42:31.285 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.280+0000 7ff8fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:31.285 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.280+0000 7ff8fe7fc640 1 -- 172.21.15.72:0/2653384946 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3375830806 0 0) 0x7ff908133c30 con 0x7ff908143c50 2024-03-29T11:42:31.285 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.280+0000 7ff8fe7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:31.285 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.280+0000 7ff8fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:31.285 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.280+0000 7ff8fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:31.285 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.280+0000 7ff8fe7fc640 10 monclient(hunting): my global_id is 4447 2024-03-29T11:42:31.285 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.280+0000 7ff8fe7fc640 1 -- 172.21.15.72:0/2653384946 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8ec0051d0 con 0x7ff908143c50 2024-03-29T11:42:31.285 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.280+0000 7ff8fe7fc640 1 -- 172.21.15.72:0/2653384946 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3119834735 0 0) 0x7ff90813f4d0 con 0x7ff9080708a0 2024-03-29T11:42:31.285 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.280+0000 7ff8fe7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:31.285 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.280+0000 7ff8fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:31.285 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.280+0000 7ff8fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:31.285 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.280+0000 7ff8fe7fc640 10 monclient(hunting): my global_id is 4494 2024-03-29T11:42:31.286 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.280+0000 7ff8fe7fc640 1 -- 172.21.15.72:0/2653384946 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff908133c30 con 0x7ff9080708a0 2024-03-29T11:42:31.286 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.280+0000 7ff8fe7fc640 1 -- 172.21.15.72:0/2653384946 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2644193982 0 0) 0x7ff8ec004eb0 con 0x7ff90806fe90 2024-03-29T11:42:31.286 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.281+0000 7ff8fe7fc640 1 -- 172.21.15.72:0/2653384946 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff90813f4d0 con 0x7ff90806fe90 2024-03-29T11:42:31.286 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.281+0000 7ff8fe7fc640 1 -- 172.21.15.72:0/2653384946 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7ff904001b70 con 0x7ff90806fe90 2024-03-29T11:42:31.286 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.281+0000 7ff8fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:31.286 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.281+0000 7ff8fe7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:31.286 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.281+0000 7ff8fe7fc640 10 monclient(hunting): dump: 2024-03-29T11:42:31.286 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:42:31.286 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:31.286 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:31.286 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:31.286 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:31.287 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:31.287 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:42:31.287 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:42:31.287 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:42:31.287 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:31.287 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.281+0000 7ff8fe7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:31.287 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.281+0000 7ff8fe7fc640 1 -- 172.21.15.72:0/2653384946 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2072616810 0 0) 0x7ff8ec0051d0 con 0x7ff908143c50 2024-03-29T11:42:31.287 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.281+0000 7ff8fe7fc640 1 -- 172.21.15.72:0/2653384946 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8ec004eb0 con 0x7ff908143c50 2024-03-29T11:42:31.287 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.281+0000 7ff8fe7fc640 1 -- 172.21.15.72:0/2653384946 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4009159862 0 0) 0x7ff908133c30 con 0x7ff9080708a0 2024-03-29T11:42:31.287 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.281+0000 7ff8fe7fc640 1 -- 172.21.15.72:0/2653384946 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8ec0051d0 con 0x7ff9080708a0 2024-03-29T11:42:31.287 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.281+0000 7ff8fe7fc640 1 -- 172.21.15.72:0/2653384946 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7ff900001bb0 con 0x7ff908143c50 2024-03-29T11:42:31.287 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.281+0000 7ff8fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:31.287 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.281+0000 7ff8fe7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:31.287 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.281+0000 7ff8fe7fc640 10 monclient(hunting): dump: 2024-03-29T11:42:31.287 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:42:31.288 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:31.288 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:31.288 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:31.288 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:31.288 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:31.288 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:42:31.288 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:42:31.288 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:42:31.288 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:31.289 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.281+0000 7ff8fe7fc640 1 -- 172.21.15.72:0/2653384946 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7ff8f8001610 con 0x7ff9080708a0 2024-03-29T11:42:31.289 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.281+0000 7ff8fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:31.289 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.281+0000 7ff8fe7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:31.289 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.281+0000 7ff8fe7fc640 10 monclient(hunting): dump: 2024-03-29T11:42:31.289 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:42:31.289 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:31.289 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:31.289 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:31.289 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:31.289 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:31.289 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:42:31.289 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:42:31.289 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:42:31.289 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:31.289 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.281+0000 7ff8fe7fc640 1 -- 172.21.15.72:0/2653384946 <== mon.2 v1:172.21.15.72:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3624002466 0 0) 0x7ff90813f4d0 con 0x7ff90806fe90 2024-03-29T11:42:31.290 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.281+0000 7ff8fe7fc640 1 -- 172.21.15.72:0/2653384946 >> v1:172.21.15.89:6789/0 conn(0x7ff908143c50 legacy=0x7ff908140070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:31.290 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.281+0000 7ff8fe7fc640 1 -- 172.21.15.72:0/2653384946 >> v1:172.21.15.72:6789/0 conn(0x7ff9080708a0 legacy=0x7ff90813f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:31.290 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.281+0000 7ff8fe7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:42:31.290 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.281+0000 7ff8fe7fc640 1 monclient: found mon.c 2024-03-29T11:42:31.290 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.281+0000 7ff8fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:31.291 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.281+0000 7ff8fe7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:42:31.291 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.281+0000 7ff8fe7fc640 1 -- 172.21.15.72:0/2653384946 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9080ab380 con 0x7ff90806fe90 2024-03-29T11:42:31.291 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.281+0000 7ff8fe7fc640 10 monclient: _finish_auth 0 2024-03-29T11:42:31.291 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.281+0000 7ff8fe7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:42:31.291 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.281+0000 7ff8fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:31.293 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.283+0000 7ff90fd1c640 5 monclient: authenticate success, global_id 4508 2024-03-29T11:42:31.293 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.283+0000 7ff90fd1c640 10 monclient: _renew_subs 2024-03-29T11:42:31.293 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.283+0000 7ff90fd1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:42:31.293 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.283+0000 7ff90fd1c640 1 -- 172.21.15.72:0/2653384946 --> v1:172.21.15.72:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9080ab630 con 0x7ff90806fe90 2024-03-29T11:42:31.293 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.283+0000 7ff90fd1c640 10 client.4508.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:31.293 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.283+0000 7ff90fd1c640 10 monclient: _renew_subs 2024-03-29T11:42:31.293 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.283+0000 7ff90fd1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:42:31.293 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.283+0000 7ff90fd1c640 1 -- 172.21.15.72:0/2653384946 --> v1:172.21.15.72:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9081e3700 con 0x7ff90806fe90 2024-03-29T11:42:31.293 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.283+0000 7ff90fd1c640 1 librados: init done 2024-03-29T11:42:31.294 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.285+0000 7ff8fe7fc640 1 -- 172.21.15.72:0/2653384946 <== mon.2 v1:172.21.15.72:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff904001d50 con 0x7ff90806fe90 2024-03-29T11:42:31.294 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.285+0000 7ff8fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:31.294 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.285+0000 7ff8fe7fc640 1 -- 172.21.15.72:0/2653384946 <== mon.2 v1:172.21.15.72:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7ff904004d90 con 0x7ff90806fe90 2024-03-29T11:42:31.294 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.285+0000 7ff8fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:31.294 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.285+0000 7ff8fe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:31.294 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.285+0000 7ff8fe7fc640 10 monclient: dump: 2024-03-29T11:42:31.294 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:42:31.294 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:31.294 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:31.295 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:31.295 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:31.295 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:31.295 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:42:31.295 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:42:31.295 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:42:31.295 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:31.295 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.285+0000 7ff90fd1c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:31.295 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.286+0000 7ff90fd1c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:31.295 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.286+0000 7ff90fd1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:42:31.295 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.286+0000 7ff90fd1c640 1 -- 172.21.15.72:0/2653384946 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff908071e40 con 0x7ff90806fe90 2024-03-29T11:42:31.296 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.287+0000 7ff8fffff640 4 set_mon_vals no callback set 2024-03-29T11:42:31.297 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.290+0000 7ff8fe7fc640 1 -- 172.21.15.72:0/2653384946 <== mon.2 v1:172.21.15.72:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7ff90401ad60 con 0x7ff90806fe90 2024-03-29T11:42:31.297 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.291+0000 7ff8fe7fc640 1 -- 172.21.15.72:0/2653384946 <== mon.2 v1:172.21.15.72:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7ff904084a70 con 0x7ff90806fe90 2024-03-29T11:42:31.297 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.291+0000 7ff8fe7fc640 10 client.4508.objecter ms_dispatch 0x7ff908000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:31.297 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.291+0000 7ff8fe7fc640 3 client.4508.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:31.297 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.291+0000 7ff8fe7fc640 3 client.4508.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:31.297 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.291+0000 7ff8fe7fc640 20 client.4508.objecter dump_active .. 0 homeless 2024-03-29T11:42:31.297 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.291+0000 7ff90d2b7640 10 monclient: get_auth_request con 0x7ff8ec06e2e0 auth_method 0 2024-03-29T11:42:31.297 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.292+0000 7ff8fe7fc640 10 client.4508.objecter ms_handle_connect 0x7ff8ec06e2e0 2024-03-29T11:42:31.298 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.295+0000 7ff8fe7fc640 1 -- 172.21.15.72:0/2653384946 <== mon.2 v1:172.21.15.72:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff904050350 con 0x7ff90806fe90 2024-03-29T11:42:31.298 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.295+0000 7ff8fe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:31.298 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.295+0000 7ff8fe7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:31.627 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.624+0000 7ff90fd1c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:31.628 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.624+0000 7ff90fd1c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:31.628 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.624+0000 7ff90fd1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:42:31.628 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.624+0000 7ff90fd1c640 1 -- 172.21.15.72:0/2653384946 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff90813f680 con 0x7ff90806fe90 2024-03-29T11:42:31.628 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.625+0000 7ff8fe7fc640 1 -- 172.21.15.72:0/2653384946 <== mon.2 v1:172.21.15.72:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7ff90813f680 con 0x7ff90806fe90 2024-03-29T11:42:31.628 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.625+0000 7ff8fe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:31.628 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.625+0000 7ff8fe7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:31.630 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:42:31.630 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:31.632 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.629+0000 7ff90fd1c640 10 librados: watch_flush enter 2024-03-29T11:42:31.634 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.631+0000 7ff90fd1c640 10 librados: watch_flush exit 2024-03-29T11:42:31.634 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.631+0000 7ff90fd1c640 20 client.4508.objecter shutdown clearing up homeless session... 2024-03-29T11:42:31.634 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.631+0000 7ff90fd1c640 10 client.4508.objecter successfully canceled tick 2024-03-29T11:42:31.635 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.631+0000 7ff90fd1c640 1 -- 172.21.15.72:0/2653384946 >> v1:172.21.15.72:6816/1578866102 conn(0x7ff8ec06e2e0 legacy=0x7ff8ec0707a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:31.635 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.631+0000 7ff90fd1c640 10 monclient: shutdown 2024-03-29T11:42:31.635 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.631+0000 7ff90fd1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:31.635 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.631+0000 7ff90fd1c640 1 -- 172.21.15.72:0/2653384946 >> v1:172.21.15.72:6790/0 conn(0x7ff90806fe90 legacy=0x7ff9080aa4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:31.635 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.631+0000 7ff90fd1c640 1 -- 172.21.15.72:0/2653384946 shutdown_connections 2024-03-29T11:42:31.635 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.632+0000 7ff90fd1c640 1 -- 172.21.15.72:0/2653384946 >> 172.21.15.72:0/2653384946 conn(0x7ff9080675b0 msgr2=0x7ff90806d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:31.635 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.632+0000 7ff90fd1c640 1 -- 172.21.15.72:0/2653384946 shutdown_connections 2024-03-29T11:42:31.635 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.632+0000 7ff90fd1c640 1 -- 172.21.15.72:0/2653384946 wait complete. 2024-03-29T11:42:31.635 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:31.632+0000 7ff90fd1c640 1 librados: shutdown 2024-03-29T11:42:31.651 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:38.459 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:38.611 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.608+0000 7fca36c0e640 10 monclient: get_monmap_and_config 2024-03-29T11:42:38.612 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.608+0000 7fca36c0e640 10 monclient: build_initial_monmap 2024-03-29T11:42:38.612 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.608+0000 7fca36c0e640 10 monclient: monmap: 2024-03-29T11:42:38.612 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:42:38.612 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:38.612 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:42:38.609399+0000 2024-03-29T11:42:38.612 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:42:38.609399+0000 2024-03-29T11:42:38.612 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:38.612 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:38.612 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:42:38.612 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:42:38.612 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:42:38.612 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:38.613 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.609+0000 7fca36c0e640 1 Processor -- start 2024-03-29T11:42:38.613 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.609+0000 7fca36c0e640 1 -- start start 2024-03-29T11:42:38.613 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.609+0000 7fca36c0e640 10 monclient: init 2024-03-29T11:42:38.613 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.610+0000 7fca36c0e640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:38.614 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.610+0000 7fca36c0e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:42:38.614 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.610+0000 7fca36c0e640 10 monclient(hunting): picked mon.noname-b con 0x7fca301457f0 addr v1:172.21.15.72:6790/0 2024-03-29T11:42:38.614 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.610+0000 7fca36c0e640 10 monclient(hunting): picked mon.noname-c con 0x7fca3013f650 addr v1:172.21.15.89:6789/0 2024-03-29T11:42:38.614 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.610+0000 7fca36c0e640 10 monclient(hunting): picked mon.noname-a con 0x7fca300a9d70 addr v1:172.21.15.72:6789/0 2024-03-29T11:42:38.614 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.610+0000 7fca36c0e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca300aaef0 con 0x7fca300a9d70 2024-03-29T11:42:38.614 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.610+0000 7fca36c0e640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca30145bd0 con 0x7fca3013f650 2024-03-29T11:42:38.614 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.610+0000 7fca36c0e640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca30145eb0 con 0x7fca301457f0 2024-03-29T11:42:38.614 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.610+0000 7fca36c0e640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:38.614 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.610+0000 7fca36c0e640 10 monclient(hunting): authenticate will time out at 1317.898752s 2024-03-29T11:42:38.614 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.610+0000 7fca351ab640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fca300a9d70 0x7fca3014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:33992/0 (socket says 172.21.15.72:33992) 2024-03-29T11:42:38.614 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.610+0000 7fca351ab640 1 -- 172.21.15.72:0/245971642 learned_addr learned my addr 172.21.15.72:0/245971642 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:42:38.615 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.611+0000 7fca2f7fe640 1 -- 172.21.15.72:0/245971642 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1226005816 0 0) 0x7fca30145bd0 con 0x7fca3013f650 2024-03-29T11:42:38.616 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.611+0000 7fca2f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:38.616 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.611+0000 7fca2f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:38.616 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.611+0000 7fca2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:38.616 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.611+0000 7fca2f7fe640 10 monclient(hunting): my global_id is 4450 2024-03-29T11:42:38.616 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.611+0000 7fca2f7fe640 1 -- 172.21.15.72:0/245971642 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca18003520 con 0x7fca3013f650 2024-03-29T11:42:38.616 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.611+0000 7fca2f7fe640 1 -- 172.21.15.72:0/245971642 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2553844574 0 0) 0x7fca30145eb0 con 0x7fca301457f0 2024-03-29T11:42:38.616 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.611+0000 7fca2f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:38.616 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.611+0000 7fca2f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:38.616 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.611+0000 7fca2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:38.616 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.611+0000 7fca2f7fe640 10 monclient(hunting): my global_id is 4511 2024-03-29T11:42:38.616 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.611+0000 7fca2f7fe640 1 -- 172.21.15.72:0/245971642 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca30145bd0 con 0x7fca301457f0 2024-03-29T11:42:38.616 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.612+0000 7fca2f7fe640 1 -- 172.21.15.72:0/245971642 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4182740336 0 0) 0x7fca300aaef0 con 0x7fca300a9d70 2024-03-29T11:42:38.616 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.612+0000 7fca2f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:38.616 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.612+0000 7fca2f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:38.616 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.612+0000 7fca2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:38.617 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.612+0000 7fca2f7fe640 10 monclient(hunting): my global_id is 4497 2024-03-29T11:42:38.617 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.612+0000 7fca2f7fe640 1 -- 172.21.15.72:0/245971642 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca30145eb0 con 0x7fca300a9d70 2024-03-29T11:42:38.617 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.612+0000 7fca2f7fe640 1 -- 172.21.15.72:0/245971642 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1393059236 0 0) 0x7fca18003520 con 0x7fca3013f650 2024-03-29T11:42:38.617 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.612+0000 7fca2f7fe640 1 -- 172.21.15.72:0/245971642 >> v1:172.21.15.72:6790/0 conn(0x7fca301457f0 legacy=0x7fca3013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:38.617 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.612+0000 7fca2f7fe640 1 -- 172.21.15.72:0/245971642 >> v1:172.21.15.72:6789/0 conn(0x7fca300a9d70 legacy=0x7fca3014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:38.617 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.612+0000 7fca2f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:42:38.617 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.612+0000 7fca2f7fe640 1 monclient: found mon.noname-c 2024-03-29T11:42:38.617 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.612+0000 7fca2f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:38.617 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.612+0000 7fca2f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.89:6789/0 2024-03-29T11:42:38.617 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.612+0000 7fca2f7fe640 1 -- 172.21.15.72:0/245971642 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca30146340 con 0x7fca3013f650 2024-03-29T11:42:38.617 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.612+0000 7fca2f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:42:38.617 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.612+0000 7fca2f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:42:38.617 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.612+0000 7fca2f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:38.617 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.612+0000 7fca2f7fe640 1 -- 172.21.15.72:0/245971642 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fca24001690 con 0x7fca3013f650 2024-03-29T11:42:38.618 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.612+0000 7fca2f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:38.618 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.612+0000 7fca2f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:38.618 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.612+0000 7fca2f7fe640 10 monclient: dump: 2024-03-29T11:42:38.618 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:42:38.618 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:38.618 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:38.619 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:38.619 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:38.619 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:38.619 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:42:38.619 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:42:38.620 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:42:38.620 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:38.620 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.612+0000 7fca36c0e640 5 monclient: authenticate success, global_id 4450 2024-03-29T11:42:38.620 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.612+0000 7fca36c0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:38.620 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.613+0000 7fca2f7fe640 1 -- 172.21.15.72:0/245971642 <== mon.1 v1:172.21.15.89:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca240017f0 con 0x7fca3013f650 2024-03-29T11:42:38.620 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.613+0000 7fca2f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:38.620 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.613+0000 7fca2f7fe640 1 -- 172.21.15.72:0/245971642 <== mon.1 v1:172.21.15.89:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fca24001c00 con 0x7fca3013f650 2024-03-29T11:42:38.620 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.613+0000 7fca2f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:38.621 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.613+0000 7fca2f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:38.621 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.613+0000 7fca2f7fe640 10 monclient: dump: 2024-03-29T11:42:38.621 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:42:38.621 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:38.621 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:38.621 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:38.621 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:38.621 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:38.621 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:42:38.621 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:42:38.621 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:42:38.621 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:38.621 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.613+0000 7fca36c0e640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:38.621 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.613+0000 7fca36c0e640 4 set_mon_vals no callback set 2024-03-29T11:42:38.632 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.614+0000 7fca36c0e640 10 monclient: shutdown 2024-03-29T11:42:38.632 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.614+0000 7fca36c0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:38.632 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.614+0000 7fca36c0e640 1 -- 172.21.15.72:0/245971642 >> v1:172.21.15.89:6789/0 conn(0x7fca3013f650 legacy=0x7fca300b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:38.632 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.614+0000 7fca36c0e640 1 -- 172.21.15.72:0/245971642 shutdown_connections 2024-03-29T11:42:38.632 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.614+0000 7fca36c0e640 1 -- 172.21.15.72:0/245971642 >> 172.21.15.72:0/245971642 conn(0x7fca30066a00 msgr2=0x7fca30066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:38.632 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.614+0000 7fca36c0e640 1 -- 172.21.15.72:0/245971642 shutdown_connections 2024-03-29T11:42:38.632 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.614+0000 7fca36c0e640 1 -- 172.21.15.72:0/245971642 wait complete. 2024-03-29T11:42:38.632 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.615+0000 7fca36c0e640 10 monclient: build_initial_monmap 2024-03-29T11:42:38.632 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.615+0000 7fca36c0e640 10 monclient: monmap: 2024-03-29T11:42:38.633 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:42:38.633 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:38.633 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:42:38.633 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:42:38.633 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:38.633 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:38.633 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:42:38.633 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:42:38.633 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:42:38.633 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:38.633 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.616+0000 7fca36c0e640 1 librados: starting msgr at 2024-03-29T11:42:38.633 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.616+0000 7fca36c0e640 1 librados: starting objecter 2024-03-29T11:42:38.633 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.616+0000 7fca36c0e640 1 Processor -- start 2024-03-29T11:42:38.633 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.616+0000 7fca36c0e640 1 -- start start 2024-03-29T11:42:38.634 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.616+0000 7fca36c0e640 1 librados: setting wanted keys 2024-03-29T11:42:38.634 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.616+0000 7fca36c0e640 1 librados: calling monclient init 2024-03-29T11:42:38.634 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.616+0000 7fca36c0e640 10 monclient: init 2024-03-29T11:42:38.634 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.617+0000 7fca36c0e640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:38.634 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.617+0000 7fca36c0e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:42:38.634 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.618+0000 7fca36c0e640 10 monclient(hunting): picked mon.noname-c con 0x7fca300a9d70 addr v1:172.21.15.72:6790/0 2024-03-29T11:42:38.634 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.618+0000 7fca36c0e640 10 monclient(hunting): picked mon.noname-b con 0x7fca3013f650 addr v1:172.21.15.89:6789/0 2024-03-29T11:42:38.635 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.618+0000 7fca36c0e640 10 monclient(hunting): picked mon.noname-a con 0x7fca301457f0 addr v1:172.21.15.72:6789/0 2024-03-29T11:42:38.635 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.618+0000 7fca36c0e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca301cf030 con 0x7fca301457f0 2024-03-29T11:42:38.635 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.618+0000 7fca36c0e640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca301399b0 con 0x7fca3013f650 2024-03-29T11:42:38.635 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.618+0000 7fca351ab640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fca301457f0 0x7fca301392a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:34700/0 (socket says 172.21.15.72:34700) 2024-03-29T11:42:38.635 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.618+0000 7fca351ab640 1 -- 172.21.15.72:0/56317248 learned_addr learned my addr 172.21.15.72:0/56317248 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:42:38.635 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.618+0000 7fca2d7fa640 10 client.?.objecter ms_handle_connect 0x7fca301457f0 2024-03-29T11:42:38.635 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.618+0000 7fca2d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:38.635 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.618+0000 7fca36c0e640 1 -- 172.21.15.72:0/56317248 --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca3013a160 con 0x7fca300a9d70 2024-03-29T11:42:38.636 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.618+0000 7fca2d7fa640 10 client.?.objecter ms_handle_connect 0x7fca300a9d70 2024-03-29T11:42:38.636 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.618+0000 7fca2d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:38.636 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.618+0000 7fca36c0e640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:38.636 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.618+0000 7fca36c0e640 10 monclient(hunting): authenticate will time out at 1317.906977s 2024-03-29T11:42:38.636 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.618+0000 7fca2d7fa640 10 client.?.objecter ms_handle_connect 0x7fca3013f650 2024-03-29T11:42:38.636 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.619+0000 7fca2d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:38.636 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.619+0000 7fca2d7fa640 1 -- 172.21.15.72:0/56317248 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2768710675 0 0) 0x7fca301cf030 con 0x7fca301457f0 2024-03-29T11:42:38.637 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.619+0000 7fca2d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:42:38.637 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.619+0000 7fca2d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:38.637 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.619+0000 7fca2d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:38.637 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.619+0000 7fca2d7fa640 10 monclient(hunting): my global_id is 4500 2024-03-29T11:42:38.637 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.619+0000 7fca2d7fa640 1 -- 172.21.15.72:0/56317248 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca1c004eb0 con 0x7fca301457f0 2024-03-29T11:42:38.637 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.619+0000 7fca2d7fa640 1 -- 172.21.15.72:0/56317248 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2840391610 0 0) 0x7fca301399b0 con 0x7fca3013f650 2024-03-29T11:42:38.637 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.619+0000 7fca2d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:42:38.638 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.619+0000 7fca2d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:38.638 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.619+0000 7fca2d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:38.638 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.619+0000 7fca2d7fa640 10 monclient(hunting): my global_id is 4453 2024-03-29T11:42:38.638 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.619+0000 7fca2d7fa640 1 -- 172.21.15.72:0/56317248 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca301cf030 con 0x7fca3013f650 2024-03-29T11:42:38.638 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.619+0000 7fca2d7fa640 1 -- 172.21.15.72:0/56317248 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 770758803 0 0) 0x7fca3013a160 con 0x7fca300a9d70 2024-03-29T11:42:38.638 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.619+0000 7fca2d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:42:38.638 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.619+0000 7fca2d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:38.639 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.619+0000 7fca2d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:38.639 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.619+0000 7fca2d7fa640 10 monclient(hunting): my global_id is 4514 2024-03-29T11:42:38.639 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.619+0000 7fca2d7fa640 1 -- 172.21.15.72:0/56317248 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca301399b0 con 0x7fca300a9d70 2024-03-29T11:42:38.639 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.619+0000 7fca2d7fa640 1 -- 172.21.15.72:0/56317248 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1311386882 0 0) 0x7fca1c004eb0 con 0x7fca301457f0 2024-03-29T11:42:38.639 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.619+0000 7fca2d7fa640 1 -- 172.21.15.72:0/56317248 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca3013a160 con 0x7fca301457f0 2024-03-29T11:42:38.639 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.619+0000 7fca2d7fa640 1 -- 172.21.15.72:0/56317248 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fca28001ba0 con 0x7fca301457f0 2024-03-29T11:42:38.640 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.619+0000 7fca2d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:38.640 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.619+0000 7fca2d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:38.640 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.619+0000 7fca2d7fa640 10 monclient(hunting): dump: 2024-03-29T11:42:38.640 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:42:38.640 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:38.640 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:38.640 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:38.640 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:38.640 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:38.640 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:42:38.640 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:42:38.640 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:42:38.640 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:38.640 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.619+0000 7fca2d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:38.640 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.620+0000 7fca2d7fa640 1 -- 172.21.15.72:0/56317248 <== mon.0 v1:172.21.15.72:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1346022308 0 0) 0x7fca3013a160 con 0x7fca301457f0 2024-03-29T11:42:38.641 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.620+0000 7fca2d7fa640 1 -- 172.21.15.72:0/56317248 >> v1:172.21.15.72:6790/0 conn(0x7fca300a9d70 legacy=0x7fca301cd610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:38.641 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.620+0000 7fca2d7fa640 1 -- 172.21.15.72:0/56317248 >> v1:172.21.15.89:6789/0 conn(0x7fca3013f650 legacy=0x7fca30138b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:38.641 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.620+0000 7fca2d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:42:38.641 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.620+0000 7fca2d7fa640 1 monclient: found mon.a 2024-03-29T11:42:38.641 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.620+0000 7fca2d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:38.641 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.620+0000 7fca2d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:42:38.641 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.620+0000 7fca2d7fa640 1 -- 172.21.15.72:0/56317248 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca301cb340 con 0x7fca301457f0 2024-03-29T11:42:38.641 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.620+0000 7fca2d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:42:38.641 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.620+0000 7fca2d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:42:38.641 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.620+0000 7fca2d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:38.641 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.620+0000 7fca36c0e640 5 monclient: authenticate success, global_id 4500 2024-03-29T11:42:38.641 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.620+0000 7fca36c0e640 10 monclient: _renew_subs 2024-03-29T11:42:38.641 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.620+0000 7fca36c0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:42:38.641 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.620+0000 7fca36c0e640 1 -- 172.21.15.72:0/56317248 --> v1:172.21.15.72:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca301cdd20 con 0x7fca301457f0 2024-03-29T11:42:38.641 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.620+0000 7fca36c0e640 10 client.4500.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:38.642 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.620+0000 7fca36c0e640 10 monclient: _renew_subs 2024-03-29T11:42:38.642 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.620+0000 7fca36c0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:42:38.642 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.620+0000 7fca36c0e640 1 -- 172.21.15.72:0/56317248 --> v1:172.21.15.72:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fca301cce20 con 0x7fca301457f0 2024-03-29T11:42:38.642 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.620+0000 7fca36c0e640 1 librados: init done 2024-03-29T11:42:38.642 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.620+0000 7fca2d7fa640 1 -- 172.21.15.72:0/56317248 <== mon.0 v1:172.21.15.72:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca28001d00 con 0x7fca301457f0 2024-03-29T11:42:38.642 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.620+0000 7fca2d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:38.642 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.620+0000 7fca2e7fc640 4 set_mon_vals no callback set 2024-03-29T11:42:38.642 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.620+0000 7fca2d7fa640 1 -- 172.21.15.72:0/56317248 <== mon.0 v1:172.21.15.72:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fca28004d00 con 0x7fca301457f0 2024-03-29T11:42:38.642 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.620+0000 7fca2d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:38.643 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.620+0000 7fca2d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:38.643 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.620+0000 7fca2d7fa640 10 monclient: dump: 2024-03-29T11:42:38.643 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:42:38.643 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:38.643 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:38.643 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:38.643 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:38.643 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:38.643 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:42:38.643 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:42:38.644 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:42:38.644 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:38.644 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.622+0000 7fca36c0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:38.644 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.622+0000 7fca36c0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:38.644 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.622+0000 7fca36c0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:42:38.644 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.622+0000 7fca36c0e640 1 -- 172.21.15.72:0/56317248 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca10005080 con 0x7fca301457f0 2024-03-29T11:42:38.644 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.623+0000 7fca2d7fa640 1 -- 172.21.15.72:0/56317248 <== mon.0 v1:172.21.15.72:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7fca28004e60 con 0x7fca301457f0 2024-03-29T11:42:38.644 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.624+0000 7fca2d7fa640 1 -- 172.21.15.72:0/56317248 <== mon.0 v1:172.21.15.72:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7fca28084d60 con 0x7fca301457f0 2024-03-29T11:42:38.645 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.624+0000 7fca2d7fa640 10 client.4500.objecter ms_dispatch 0x7fca30000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:38.645 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.624+0000 7fca2d7fa640 3 client.4500.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:38.645 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.624+0000 7fca2d7fa640 3 client.4500.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:38.645 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.624+0000 7fca2d7fa640 20 client.4500.objecter dump_active .. 0 homeless 2024-03-29T11:42:38.645 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.624+0000 7fca349aa640 10 monclient: get_auth_request con 0x7fca1c06de30 auth_method 0 2024-03-29T11:42:38.645 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.625+0000 7fca2d7fa640 10 client.4500.objecter ms_handle_connect 0x7fca1c06de30 2024-03-29T11:42:38.646 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.633+0000 7fca2d7fa640 1 -- 172.21.15.72:0/56317248 <== mon.0 v1:172.21.15.72:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fca280506a0 con 0x7fca301457f0 2024-03-29T11:42:38.646 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.633+0000 7fca2d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:38.646 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:38.633+0000 7fca2d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:39.013 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:39.010+0000 7fca36c0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:39.013 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:39.010+0000 7fca36c0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:39.013 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:39.010+0000 7fca36c0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:42:39.014 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:39.010+0000 7fca36c0e640 1 -- 172.21.15.72:0/56317248 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca10005330 con 0x7fca301457f0 2024-03-29T11:42:39.014 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:39.011+0000 7fca2d7fa640 1 -- 172.21.15.72:0/56317248 <== mon.0 v1:172.21.15.72:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7fca280525d0 con 0x7fca301457f0 2024-03-29T11:42:39.014 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:39.011+0000 7fca2d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:39.014 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:39.011+0000 7fca2d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:39.015 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:42:39.015 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:39.018 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:39.015+0000 7fca36c0e640 10 librados: watch_flush enter 2024-03-29T11:42:39.018 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:39.015+0000 7fca36c0e640 10 librados: watch_flush exit 2024-03-29T11:42:39.018 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:39.015+0000 7fca36c0e640 20 client.4500.objecter shutdown clearing up homeless session... 2024-03-29T11:42:39.018 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:39.015+0000 7fca36c0e640 10 client.4500.objecter successfully canceled tick 2024-03-29T11:42:39.018 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:39.015+0000 7fca36c0e640 1 -- 172.21.15.72:0/56317248 >> v1:172.21.15.72:6816/1578866102 conn(0x7fca1c06de30 legacy=0x7fca1c0702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:39.018 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:39.015+0000 7fca36c0e640 10 monclient: shutdown 2024-03-29T11:42:39.018 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:39.015+0000 7fca36c0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:39.019 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:39.015+0000 7fca36c0e640 1 -- 172.21.15.72:0/56317248 >> v1:172.21.15.72:6789/0 conn(0x7fca301457f0 legacy=0x7fca301392a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:39.019 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:39.015+0000 7fca36c0e640 1 -- 172.21.15.72:0/56317248 shutdown_connections 2024-03-29T11:42:39.020 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:39.015+0000 7fca36c0e640 1 -- 172.21.15.72:0/56317248 >> 172.21.15.72:0/56317248 conn(0x7fca30066a00 msgr2=0x7fca3013aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:39.020 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:39.016+0000 7fca36c0e640 1 -- 172.21.15.72:0/56317248 shutdown_connections 2024-03-29T11:42:39.020 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:39.016+0000 7fca36c0e640 1 -- 172.21.15.72:0/56317248 wait complete. 2024-03-29T11:42:39.020 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:39.016+0000 7fca36c0e640 1 librados: shutdown 2024-03-29T11:42:39.033 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:45.840 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:45.992 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.989+0000 7fc5cca7b640 10 monclient: get_monmap_and_config 2024-03-29T11:42:45.992 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.989+0000 7fc5cca7b640 10 monclient: build_initial_monmap 2024-03-29T11:42:45.992 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.989+0000 7fc5cca7b640 10 monclient: monmap: 2024-03-29T11:42:45.992 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:42:45.992 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:45.992 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:42:45.990008+0000 2024-03-29T11:42:45.993 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:42:45.990008+0000 2024-03-29T11:42:45.993 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:45.993 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:45.993 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:42:45.993 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:42:45.993 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:42:45.993 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:45.993 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.990+0000 7fc5cca7b640 1 Processor -- start 2024-03-29T11:42:45.993 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.990+0000 7fc5cca7b640 1 -- start start 2024-03-29T11:42:45.994 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.990+0000 7fc5cca7b640 10 monclient: init 2024-03-29T11:42:45.996 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.991+0000 7fc5cca7b640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:45.996 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.991+0000 7fc5cca7b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:42:45.996 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.991+0000 7fc5cca7b640 10 monclient(hunting): picked mon.noname-a con 0x7fc5c80b0860 addr v1:172.21.15.72:6789/0 2024-03-29T11:42:45.996 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.991+0000 7fc5cca7b640 10 monclient(hunting): picked mon.noname-b con 0x7fc5c813ef30 addr v1:172.21.15.72:6790/0 2024-03-29T11:42:45.997 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.991+0000 7fc5cca7b640 10 monclient(hunting): picked mon.noname-c con 0x7fc5c80af8e0 addr v1:172.21.15.89:6789/0 2024-03-29T11:42:45.997 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.991+0000 7fc5cca7b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c80b0c40 con 0x7fc5c80b0860 2024-03-29T11:42:45.997 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.991+0000 7fc5cca7b640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c80aed10 con 0x7fc5c80af8e0 2024-03-29T11:42:45.997 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.991+0000 7fc5cca7b640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c80afd50 con 0x7fc5c813ef30 2024-03-29T11:42:45.997 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.991+0000 7fc5cca7b640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:45.997 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.991+0000 7fc5cca7b640 10 monclient(hunting): authenticate will time out at 1325.279963s 2024-03-29T11:42:45.997 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.992+0000 7fc5c659c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fc5c80b0860 0x7fc5c813e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50774/0 (socket says 172.21.15.72:50774) 2024-03-29T11:42:45.997 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.992+0000 7fc5c659c640 1 -- 172.21.15.72:0/1896606731 learned_addr learned my addr 172.21.15.72:0/1896606731 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:42:45.997 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.992+0000 7fc5c559a640 1 -- 172.21.15.72:0/1896606731 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1073409680 0 0) 0x7fc5c80b0c40 con 0x7fc5c80b0860 2024-03-29T11:42:45.997 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.992+0000 7fc5c559a640 10 monclient(hunting): _negotiate 2024-03-29T11:42:45.997 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.992+0000 7fc5c559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:45.997 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.992+0000 7fc5c559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:45.997 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.992+0000 7fc5c559a640 10 monclient(hunting): my global_id is 4503 2024-03-29T11:42:45.997 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.992+0000 7fc5c559a640 1 -- 172.21.15.72:0/1896606731 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5b0003520 con 0x7fc5c80b0860 2024-03-29T11:42:45.997 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.993+0000 7fc5c559a640 1 -- 172.21.15.72:0/1896606731 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 468597948 0 0) 0x7fc5c80afd50 con 0x7fc5c813ef30 2024-03-29T11:42:45.998 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.993+0000 7fc5c559a640 10 monclient(hunting): _negotiate 2024-03-29T11:42:45.998 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.993+0000 7fc5c559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:45.998 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.993+0000 7fc5c559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:45.998 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.993+0000 7fc5c559a640 10 monclient(hunting): my global_id is 4517 2024-03-29T11:42:45.998 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.993+0000 7fc5c559a640 1 -- 172.21.15.72:0/1896606731 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5c80b0c40 con 0x7fc5c813ef30 2024-03-29T11:42:45.998 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.993+0000 7fc5c559a640 1 -- 172.21.15.72:0/1896606731 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 763539572 0 0) 0x7fc5b0003520 con 0x7fc5c80b0860 2024-03-29T11:42:45.998 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.993+0000 7fc5c559a640 1 -- 172.21.15.72:0/1896606731 >> v1:172.21.15.72:6790/0 conn(0x7fc5c813ef30 legacy=0x7fc5c80af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:45.998 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.993+0000 7fc5c559a640 1 -- 172.21.15.72:0/1896606731 >> v1:172.21.15.89:6789/0 conn(0x7fc5c80af8e0 legacy=0x7fc5c8149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:45.998 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.993+0000 7fc5c559a640 10 monclient: _finish_hunting 0 2024-03-29T11:42:45.998 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.993+0000 7fc5c559a640 1 monclient: found mon.noname-a 2024-03-29T11:42:45.998 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.993+0000 7fc5c559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:45.998 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.993+0000 7fc5c559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.72:6789/0 2024-03-29T11:42:45.998 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.993+0000 7fc5c559a640 1 -- 172.21.15.72:0/1896606731 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c80ab1b0 con 0x7fc5c80b0860 2024-03-29T11:42:45.998 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.993+0000 7fc5c559a640 10 monclient: _finish_auth 0 2024-03-29T11:42:45.998 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.993+0000 7fc5c559a640 10 monclient: _check_auth_tickets 2024-03-29T11:42:45.999 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.993+0000 7fc5c559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:45.999 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.993+0000 7fc5c559a640 1 -- 172.21.15.72:0/1896606731 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fc5b8004c90 con 0x7fc5c80b0860 2024-03-29T11:42:45.999 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.993+0000 7fc5c559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:45.999 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.993+0000 7fc5c559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:45.999 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.993+0000 7fc5c559a640 10 monclient: dump: 2024-03-29T11:42:45.999 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:42:45.999 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:45.999 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:45.999 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:45.999 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:45.999 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:45.999 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:42:45.999 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:42:45.999 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:42:45.999 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:45.999 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.993+0000 7fc5cca7b640 5 monclient: authenticate success, global_id 4503 2024-03-29T11:42:46.000 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.993+0000 7fc5cca7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:46.000 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.993+0000 7fc5c559a640 1 -- 172.21.15.72:0/1896606731 <== mon.0 v1:172.21.15.72:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5b8004e70 con 0x7fc5c80b0860 2024-03-29T11:42:46.000 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.993+0000 7fc5c559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:46.000 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.993+0000 7fc5c559a640 1 -- 172.21.15.72:0/1896606731 <== mon.0 v1:172.21.15.72:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fc5b8001410 con 0x7fc5c80b0860 2024-03-29T11:42:46.000 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.993+0000 7fc5c559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:46.000 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.993+0000 7fc5c559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:46.000 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.993+0000 7fc5c559a640 10 monclient: dump: 2024-03-29T11:42:46.000 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:42:46.000 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:46.000 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:46.000 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:46.000 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:46.000 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:46.000 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:42:46.000 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:42:46.000 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:42:46.001 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:46.001 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.993+0000 7fc5cca7b640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:46.001 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.993+0000 7fc5cca7b640 4 set_mon_vals no callback set 2024-03-29T11:42:46.006 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.994+0000 7fc5cca7b640 10 monclient: shutdown 2024-03-29T11:42:46.006 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.994+0000 7fc5cca7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:46.006 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.994+0000 7fc5cca7b640 1 -- 172.21.15.72:0/1896606731 >> v1:172.21.15.72:6789/0 conn(0x7fc5c80b0860 legacy=0x7fc5c813e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:46.006 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.994+0000 7fc5cca7b640 1 -- 172.21.15.72:0/1896606731 shutdown_connections 2024-03-29T11:42:46.006 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.994+0000 7fc5cca7b640 1 -- 172.21.15.72:0/1896606731 >> 172.21.15.72:0/1896606731 conn(0x7fc5c806b660 msgr2=0x7fc5c806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:46.007 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.994+0000 7fc5cca7b640 1 -- 172.21.15.72:0/1896606731 shutdown_connections 2024-03-29T11:42:46.007 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.994+0000 7fc5cca7b640 1 -- 172.21.15.72:0/1896606731 wait complete. 2024-03-29T11:42:46.007 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.995+0000 7fc5cca7b640 10 monclient: build_initial_monmap 2024-03-29T11:42:46.007 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.995+0000 7fc5cca7b640 10 monclient: monmap: 2024-03-29T11:42:46.007 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:42:46.007 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:46.007 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:42:46.007 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:42:46.007 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:46.007 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:46.007 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:42:46.007 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:42:46.007 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:42:46.007 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:46.007 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.996+0000 7fc5cca7b640 1 librados: starting msgr at 2024-03-29T11:42:46.007 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.996+0000 7fc5cca7b640 1 librados: starting objecter 2024-03-29T11:42:46.008 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.996+0000 7fc5cca7b640 1 Processor -- start 2024-03-29T11:42:46.008 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.996+0000 7fc5cca7b640 1 -- start start 2024-03-29T11:42:46.008 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.996+0000 7fc5cca7b640 1 librados: setting wanted keys 2024-03-29T11:42:46.008 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.996+0000 7fc5cca7b640 1 librados: calling monclient init 2024-03-29T11:42:46.008 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.996+0000 7fc5cca7b640 10 monclient: init 2024-03-29T11:42:46.008 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.997+0000 7fc5cca7b640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:46.008 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.997+0000 7fc5cca7b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:42:46.008 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.997+0000 7fc5cca7b640 10 monclient(hunting): picked mon.noname-c con 0x7fc5c80af8e0 addr v1:172.21.15.72:6790/0 2024-03-29T11:42:46.008 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.997+0000 7fc5cca7b640 10 monclient(hunting): picked mon.noname-a con 0x7fc5c80b0860 addr v1:172.21.15.72:6789/0 2024-03-29T11:42:46.008 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.997+0000 7fc5cca7b640 10 monclient(hunting): picked mon.noname-b con 0x7fc5c813ef30 addr v1:172.21.15.89:6789/0 2024-03-29T11:42:46.008 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.997+0000 7fc5cca7b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c8137e60 con 0x7fc5c80b0860 2024-03-29T11:42:46.009 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.997+0000 7fc5cca7b640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c8136ba0 con 0x7fc5c813ef30 2024-03-29T11:42:46.009 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.997+0000 7fc5cca7b640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c81367e0 con 0x7fc5c80af8e0 2024-03-29T11:42:46.009 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.997+0000 7fc5cca7b640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:46.009 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.997+0000 7fc5cca7b640 10 monclient(hunting): authenticate will time out at 1325.285688s 2024-03-29T11:42:46.009 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.997+0000 7fc5c659c640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7fc5c80af8e0 0x7fc5c8135a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:54022/0 (socket says 172.21.15.72:54022) 2024-03-29T11:42:46.009 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.997+0000 7fc5c659c640 1 -- 172.21.15.72:0/3742774460 learned_addr learned my addr 172.21.15.72:0/3742774460 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:42:46.009 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.997+0000 7fc5b6ffd640 10 client.?.objecter ms_handle_connect 0x7fc5c80af8e0 2024-03-29T11:42:46.009 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.997+0000 7fc5b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:46.009 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.997+0000 7fc5b6ffd640 10 client.?.objecter ms_handle_connect 0x7fc5c80b0860 2024-03-29T11:42:46.009 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.997+0000 7fc5b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:46.009 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.998+0000 7fc5b6ffd640 1 -- 172.21.15.72:0/3742774460 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2496174672 0 0) 0x7fc5c8137e60 con 0x7fc5c80b0860 2024-03-29T11:42:46.009 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.998+0000 7fc5b6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:46.009 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.998+0000 7fc5b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:46.009 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.998+0000 7fc5b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:46.009 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.998+0000 7fc5b6ffd640 10 monclient(hunting): my global_id is 4506 2024-03-29T11:42:46.010 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.998+0000 7fc5b6ffd640 1 -- 172.21.15.72:0/3742774460 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5a8004eb0 con 0x7fc5c80b0860 2024-03-29T11:42:46.010 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.998+0000 7fc5b6ffd640 1 -- 172.21.15.72:0/3742774460 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3274118754 0 0) 0x7fc5c81367e0 con 0x7fc5c80af8e0 2024-03-29T11:42:46.010 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.998+0000 7fc5b6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:46.010 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.998+0000 7fc5b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:46.010 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.998+0000 7fc5b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:46.010 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.998+0000 7fc5b6ffd640 10 monclient(hunting): my global_id is 4520 2024-03-29T11:42:46.010 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.998+0000 7fc5b6ffd640 1 -- 172.21.15.72:0/3742774460 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5c8137e60 con 0x7fc5c80af8e0 2024-03-29T11:42:46.010 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.998+0000 7fc5b6ffd640 10 client.?.objecter ms_handle_connect 0x7fc5c813ef30 2024-03-29T11:42:46.010 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.998+0000 7fc5b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:46.010 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.999+0000 7fc5b6ffd640 1 -- 172.21.15.72:0/3742774460 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1429738771 0 0) 0x7fc5a8004eb0 con 0x7fc5c80b0860 2024-03-29T11:42:46.010 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.999+0000 7fc5b6ffd640 1 -- 172.21.15.72:0/3742774460 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5c81367e0 con 0x7fc5c80b0860 2024-03-29T11:42:46.010 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.999+0000 7fc5b6ffd640 1 -- 172.21.15.72:0/3742774460 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1701416322 0 0) 0x7fc5c8136ba0 con 0x7fc5c813ef30 2024-03-29T11:42:46.010 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.999+0000 7fc5b6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:46.011 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.999+0000 7fc5b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:46.011 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.999+0000 7fc5b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:46.011 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.999+0000 7fc5b6ffd640 10 monclient(hunting): my global_id is 4456 2024-03-29T11:42:46.011 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.999+0000 7fc5b6ffd640 1 -- 172.21.15.72:0/3742774460 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5a8004eb0 con 0x7fc5c813ef30 2024-03-29T11:42:46.011 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.999+0000 7fc5b6ffd640 1 -- 172.21.15.72:0/3742774460 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fc5bc001810 con 0x7fc5c80b0860 2024-03-29T11:42:46.011 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.999+0000 7fc5b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:46.011 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.999+0000 7fc5b6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:46.011 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.999+0000 7fc5b6ffd640 10 monclient(hunting): dump: 2024-03-29T11:42:46.011 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:42:46.011 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:46.011 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:46.011 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:46.011 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:46.011 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:46.011 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:42:46.012 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:42:46.012 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:42:46.012 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:46.012 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.999+0000 7fc5b6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:46.012 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.999+0000 7fc5b6ffd640 1 -- 172.21.15.72:0/3742774460 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1924485478 0 0) 0x7fc5c8137e60 con 0x7fc5c80af8e0 2024-03-29T11:42:46.012 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.999+0000 7fc5b6ffd640 1 -- 172.21.15.72:0/3742774460 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5c8136ba0 con 0x7fc5c80af8e0 2024-03-29T11:42:46.012 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.999+0000 7fc5b6ffd640 1 -- 172.21.15.72:0/3742774460 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fc5b8004c00 con 0x7fc5c80af8e0 2024-03-29T11:42:46.012 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.999+0000 7fc5b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:46.012 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.999+0000 7fc5b6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:46.012 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.999+0000 7fc5b6ffd640 10 monclient(hunting): dump: 2024-03-29T11:42:46.012 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:42:46.012 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:46.012 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:46.012 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:46.012 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:46.013 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:46.013 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:42:46.013 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:42:46.013 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:42:46.013 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:46.013 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.999+0000 7fc5b6ffd640 1 -- 172.21.15.72:0/3742774460 <== mon.2 v1:172.21.15.72:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3789465463 0 0) 0x7fc5c8136ba0 con 0x7fc5c80af8e0 2024-03-29T11:42:46.013 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.999+0000 7fc5b6ffd640 1 -- 172.21.15.72:0/3742774460 >> v1:172.21.15.89:6789/0 conn(0x7fc5c813ef30 legacy=0x7fc5c81d3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:46.013 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.999+0000 7fc5b6ffd640 1 -- 172.21.15.72:0/3742774460 >> v1:172.21.15.72:6789/0 conn(0x7fc5c80b0860 legacy=0x7fc5c81d3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:46.013 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.999+0000 7fc5b6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:42:46.013 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.999+0000 7fc5b6ffd640 1 monclient: found mon.c 2024-03-29T11:42:46.013 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.999+0000 7fc5b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:46.013 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.999+0000 7fc5b6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:42:46.013 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.999+0000 7fc5b6ffd640 1 -- 172.21.15.72:0/3742774460 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c8136970 con 0x7fc5c80af8e0 2024-03-29T11:42:46.013 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.999+0000 7fc5b6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:42:46.013 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.999+0000 7fc5b6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:42:46.013 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.999+0000 7fc5b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:46.014 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.999+0000 7fc5cca7b640 5 monclient: authenticate success, global_id 4520 2024-03-29T11:42:46.014 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.999+0000 7fc5cca7b640 10 monclient: _renew_subs 2024-03-29T11:42:46.014 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.999+0000 7fc5cca7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:42:46.014 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.999+0000 7fc5cca7b640 1 -- 172.21.15.72:0/3742774460 --> v1:172.21.15.72:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5c81382e0 con 0x7fc5c80af8e0 2024-03-29T11:42:46.014 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.999+0000 7fc5cca7b640 10 client.4520.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:46.014 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.999+0000 7fc5cca7b640 10 monclient: _renew_subs 2024-03-29T11:42:46.014 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.999+0000 7fc5cca7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:42:46.014 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.999+0000 7fc5cca7b640 1 -- 172.21.15.72:0/3742774460 --> v1:172.21.15.72:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5c81deb20 con 0x7fc5c80af8e0 2024-03-29T11:42:46.014 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:45.999+0000 7fc5cca7b640 1 librados: init done 2024-03-29T11:42:46.015 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:46.000+0000 7fc5b6ffd640 1 -- 172.21.15.72:0/3742774460 <== mon.2 v1:172.21.15.72:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5b8001980 con 0x7fc5c80af8e0 2024-03-29T11:42:46.015 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:46.000+0000 7fc5b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:46.015 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:46.000+0000 7fc5b6ffd640 1 -- 172.21.15.72:0/3742774460 <== mon.2 v1:172.21.15.72:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fc5b8001d10 con 0x7fc5c80af8e0 2024-03-29T11:42:46.015 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:46.000+0000 7fc5b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:46.015 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:46.000+0000 7fc5b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:46.015 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:46.000+0000 7fc5b6ffd640 10 monclient: dump: 2024-03-29T11:42:46.015 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:42:46.015 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:46.015 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:46.015 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:46.015 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:46.015 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:46.015 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:42:46.015 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:42:46.016 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:42:46.016 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:46.016 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:46.000+0000 7fc5c4d99640 4 set_mon_vals no callback set 2024-03-29T11:42:46.016 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:46.001+0000 7fc5cca7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:46.016 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:46.001+0000 7fc5cca7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:46.016 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:46.001+0000 7fc5cca7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:42:46.016 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:46.001+0000 7fc5cca7b640 1 -- 172.21.15.72:0/3742774460 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5ac005080 con 0x7fc5c80af8e0 2024-03-29T11:42:46.017 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:46.004+0000 7fc5b6ffd640 1 -- 172.21.15.72:0/3742774460 <== mon.2 v1:172.21.15.72:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7fc5b801ae50 con 0x7fc5c80af8e0 2024-03-29T11:42:46.017 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:46.005+0000 7fc5c5d9b640 10 monclient: get_auth_request con 0x7fc5a8072420 auth_method 0 2024-03-29T11:42:46.017 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:46.005+0000 7fc5b6ffd640 1 -- 172.21.15.72:0/3742774460 <== mon.2 v1:172.21.15.72:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7fc5b8084a80 con 0x7fc5c80af8e0 2024-03-29T11:42:46.017 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:46.005+0000 7fc5b6ffd640 10 client.4520.objecter ms_dispatch 0x7fc5c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:46.017 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:46.005+0000 7fc5b6ffd640 3 client.4520.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:46.017 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:46.005+0000 7fc5b6ffd640 3 client.4520.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:46.018 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:46.005+0000 7fc5b6ffd640 20 client.4520.objecter dump_active .. 0 homeless 2024-03-29T11:42:46.018 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:46.005+0000 7fc5b6ffd640 10 client.4520.objecter ms_handle_connect 0x7fc5a8072420 2024-03-29T11:42:46.018 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:46.014+0000 7fc5b6ffd640 1 -- 172.21.15.72:0/3742774460 <== mon.2 v1:172.21.15.72:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc5b80503c0 con 0x7fc5c80af8e0 2024-03-29T11:42:46.018 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:46.014+0000 7fc5b6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:46.018 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:46.014+0000 7fc5b6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:46.326 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:46.323+0000 7fc5cca7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:46.326 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:46.323+0000 7fc5cca7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:46.351 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:42:46.351 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:46.409 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:46.323+0000 7fc5cca7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:42:46.410 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:46.323+0000 7fc5cca7b640 1 -- 172.21.15.72:0/3742774460 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5ac005330 con 0x7fc5c80af8e0 2024-03-29T11:42:46.410 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:46.347+0000 7fc5b6ffd640 1 -- 172.21.15.72:0/3742774460 <== mon.2 v1:172.21.15.72:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7fc5b80522f0 con 0x7fc5c80af8e0 2024-03-29T11:42:46.410 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:46.347+0000 7fc5b6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:46.410 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:46.347+0000 7fc5b6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:46.410 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:46.351+0000 7fc5cca7b640 10 librados: watch_flush enter 2024-03-29T11:42:46.410 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:46.351+0000 7fc5cca7b640 10 librados: watch_flush exit 2024-03-29T11:42:46.410 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:46.351+0000 7fc5cca7b640 20 client.4520.objecter shutdown clearing up homeless session... 2024-03-29T11:42:46.410 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:46.351+0000 7fc5cca7b640 10 client.4520.objecter successfully canceled tick 2024-03-29T11:42:46.410 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:46.351+0000 7fc5cca7b640 1 -- 172.21.15.72:0/3742774460 >> v1:172.21.15.72:6816/1578866102 conn(0x7fc5a8072420 legacy=0x7fc5a80748e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:46.410 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:46.351+0000 7fc5cca7b640 10 monclient: shutdown 2024-03-29T11:42:46.410 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:46.351+0000 7fc5cca7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:46.410 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:46.351+0000 7fc5cca7b640 1 -- 172.21.15.72:0/3742774460 >> v1:172.21.15.72:6790/0 conn(0x7fc5c80af8e0 legacy=0x7fc5c8135a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:46.411 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:46.351+0000 7fc5cca7b640 1 -- 172.21.15.72:0/3742774460 shutdown_connections 2024-03-29T11:42:46.411 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:46.351+0000 7fc5cca7b640 1 -- 172.21.15.72:0/3742774460 >> 172.21.15.72:0/3742774460 conn(0x7fc5c806b660 msgr2=0x7fc5c813e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:46.411 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:46.351+0000 7fc5cca7b640 1 -- 172.21.15.72:0/3742774460 shutdown_connections 2024-03-29T11:42:46.411 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:46.351+0000 7fc5cca7b640 1 -- 172.21.15.72:0/3742774460 wait complete. 2024-03-29T11:42:46.411 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:46.351+0000 7fc5cca7b640 1 librados: shutdown 2024-03-29T11:42:46.425 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:53.233 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:53.379 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.376+0000 7fd941f9d640 10 monclient: get_monmap_and_config 2024-03-29T11:42:53.380 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.376+0000 7fd941f9d640 10 monclient: build_initial_monmap 2024-03-29T11:42:53.380 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.376+0000 7fd941f9d640 10 monclient: monmap: 2024-03-29T11:42:53.380 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:42:53.380 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:53.380 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:42:53.377602+0000 2024-03-29T11:42:53.380 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:42:53.377602+0000 2024-03-29T11:42:53.380 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:53.380 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:53.380 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:42:53.380 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:42:53.380 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:42:53.380 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:53.381 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.377+0000 7fd941f9d640 1 Processor -- start 2024-03-29T11:42:53.381 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.378+0000 7fd941f9d640 1 -- start start 2024-03-29T11:42:53.381 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.378+0000 7fd941f9d640 10 monclient: init 2024-03-29T11:42:53.382 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.379+0000 7fd941f9d640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:53.382 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.379+0000 7fd941f9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:42:53.382 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.379+0000 7fd941f9d640 10 monclient(hunting): picked mon.noname-b con 0x7fd93402b020 addr v1:172.21.15.72:6790/0 2024-03-29T11:42:53.382 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.379+0000 7fd941f9d640 10 monclient(hunting): picked mon.noname-a con 0x7fd9340c1d60 addr v1:172.21.15.72:6789/0 2024-03-29T11:42:53.382 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.379+0000 7fd941f9d640 10 monclient(hunting): picked mon.noname-c con 0x7fd9340c4c20 addr v1:172.21.15.89:6789/0 2024-03-29T11:42:53.383 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.379+0000 7fd941f9d640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd93402b400 con 0x7fd9340c1d60 2024-03-29T11:42:53.383 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.379+0000 7fd941f9d640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd93402c210 con 0x7fd9340c4c20 2024-03-29T11:42:53.383 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.379+0000 7fd941f9d640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9340c5070 con 0x7fd93402b020 2024-03-29T11:42:53.383 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.379+0000 7fd941f9d640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:53.383 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.379+0000 7fd941f9d640 10 monclient(hunting): authenticate will time out at 1332.667656s 2024-03-29T11:42:53.383 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.380+0000 7fd93affd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fd9340c1d60 0x7fd9340c4510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:51958/0 (socket says 172.21.15.72:51958) 2024-03-29T11:42:53.383 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.380+0000 7fd93affd640 1 -- 172.21.15.72:0/1852895046 learned_addr learned my addr 172.21.15.72:0/1852895046 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:42:53.384 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.381+0000 7fd93a7fc640 1 -- 172.21.15.72:0/1852895046 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1287132408 0 0) 0x7fd93402b400 con 0x7fd9340c1d60 2024-03-29T11:42:53.384 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.381+0000 7fd93a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:53.385 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.381+0000 7fd93a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:53.385 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.381+0000 7fd93a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:53.385 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.381+0000 7fd93a7fc640 10 monclient(hunting): my global_id is 4509 2024-03-29T11:42:53.385 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.381+0000 7fd93a7fc640 1 -- 172.21.15.72:0/1852895046 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd92c003520 con 0x7fd9340c1d60 2024-03-29T11:42:53.388 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.383+0000 7fd93a7fc640 1 -- 172.21.15.72:0/1852895046 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 540941896 0 0) 0x7fd92c003520 con 0x7fd9340c1d60 2024-03-29T11:42:53.388 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.383+0000 7fd93a7fc640 1 -- 172.21.15.72:0/1852895046 >> v1:172.21.15.72:6790/0 conn(0x7fd93402b020 legacy=0x7fd9340c1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:42:53.388 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.383+0000 7fd93a7fc640 1 -- 172.21.15.72:0/1852895046 >> v1:172.21.15.89:6789/0 conn(0x7fd9340c4c20 legacy=0x7fd9340cfe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:42:53.388 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.383+0000 7fd93a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:42:53.388 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.383+0000 7fd93a7fc640 1 monclient: found mon.noname-a 2024-03-29T11:42:53.388 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.383+0000 7fd93a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:53.388 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.383+0000 7fd93a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.72:6789/0 2024-03-29T11:42:53.388 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.383+0000 7fd93a7fc640 1 -- 172.21.15.72:0/1852895046 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd93402d550 con 0x7fd9340c1d60 2024-03-29T11:42:53.388 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.383+0000 7fd93a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:42:53.388 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.383+0000 7fd93a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:42:53.388 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.383+0000 7fd93a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:53.388 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.383+0000 7fd93a7fc640 1 -- 172.21.15.72:0/1852895046 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fd930001690 con 0x7fd9340c1d60 2024-03-29T11:42:53.388 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.383+0000 7fd93a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:53.389 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.383+0000 7fd93a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:53.389 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.383+0000 7fd93a7fc640 10 monclient: dump: 2024-03-29T11:42:53.389 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:42:53.389 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:53.389 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:53.389 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:53.389 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:53.389 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:53.389 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:42:53.389 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:42:53.390 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:42:53.390 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:53.390 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.383+0000 7fd941f9d640 5 monclient: authenticate success, global_id 4509 2024-03-29T11:42:53.390 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.383+0000 7fd941f9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:53.390 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.383+0000 7fd93a7fc640 1 -- 172.21.15.72:0/1852895046 <== mon.0 v1:172.21.15.72:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9300017f0 con 0x7fd9340c1d60 2024-03-29T11:42:53.390 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.383+0000 7fd93a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:53.390 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.383+0000 7fd93a7fc640 1 -- 172.21.15.72:0/1852895046 <== mon.0 v1:172.21.15.72:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fd930001c40 con 0x7fd9340c1d60 2024-03-29T11:42:53.390 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.383+0000 7fd93a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:53.390 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.383+0000 7fd93a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:53.390 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.383+0000 7fd93a7fc640 10 monclient: dump: 2024-03-29T11:42:53.390 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:42:53.390 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:53.390 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:53.390 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:53.390 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:53.391 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:53.391 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:42:53.391 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:42:53.391 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:42:53.391 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:53.391 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.383+0000 7fd941f9d640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:53.391 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.383+0000 7fd941f9d640 4 set_mon_vals no callback set 2024-03-29T11:42:53.392 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.386+0000 7fd941f9d640 10 monclient: shutdown 2024-03-29T11:42:53.392 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.386+0000 7fd941f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:53.393 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.386+0000 7fd941f9d640 1 -- 172.21.15.72:0/1852895046 >> v1:172.21.15.72:6789/0 conn(0x7fd9340c1d60 legacy=0x7fd9340c4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:53.393 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.386+0000 7fd941f9d640 1 -- 172.21.15.72:0/1852895046 shutdown_connections 2024-03-29T11:42:53.393 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.386+0000 7fd941f9d640 1 -- 172.21.15.72:0/1852895046 >> 172.21.15.72:0/1852895046 conn(0x7fd93403a5b0 msgr2=0x7fd934036800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:53.393 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.386+0000 7fd941f9d640 1 -- 172.21.15.72:0/1852895046 shutdown_connections 2024-03-29T11:42:53.393 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.386+0000 7fd941f9d640 1 -- 172.21.15.72:0/1852895046 wait complete. 2024-03-29T11:42:53.393 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.387+0000 7fd941f9d640 10 monclient: build_initial_monmap 2024-03-29T11:42:53.393 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.387+0000 7fd941f9d640 10 monclient: monmap: 2024-03-29T11:42:53.393 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:42:53.393 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:53.393 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:42:53.393 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:42:53.393 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:53.393 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:53.393 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:42:53.393 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:42:53.394 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:42:53.394 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:53.394 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.388+0000 7fd941f9d640 1 librados: starting msgr at 2024-03-29T11:42:53.394 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.388+0000 7fd941f9d640 1 librados: starting objecter 2024-03-29T11:42:53.394 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.388+0000 7fd941f9d640 1 Processor -- start 2024-03-29T11:42:53.394 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.388+0000 7fd941f9d640 1 -- start start 2024-03-29T11:42:53.394 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.388+0000 7fd941f9d640 1 librados: setting wanted keys 2024-03-29T11:42:53.394 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.388+0000 7fd941f9d640 1 librados: calling monclient init 2024-03-29T11:42:53.394 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.388+0000 7fd941f9d640 10 monclient: init 2024-03-29T11:42:53.394 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.389+0000 7fd941f9d640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:53.394 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.389+0000 7fd941f9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:42:53.395 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.389+0000 7fd941f9d640 10 monclient(hunting): picked mon.noname-c con 0x7fd93402b020 addr v1:172.21.15.72:6790/0 2024-03-29T11:42:53.395 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.389+0000 7fd941f9d640 10 monclient(hunting): picked mon.noname-a con 0x7fd9340c1d60 addr v1:172.21.15.72:6789/0 2024-03-29T11:42:53.395 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.389+0000 7fd941f9d640 10 monclient(hunting): picked mon.noname-b con 0x7fd9340c4c20 addr v1:172.21.15.89:6789/0 2024-03-29T11:42:53.395 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.389+0000 7fd941f9d640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd934030650 con 0x7fd9340c1d60 2024-03-29T11:42:53.395 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.389+0000 7fd941f9d640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9340bb970 con 0x7fd9340c4c20 2024-03-29T11:42:53.395 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.389+0000 7fd941f9d640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd934030ae0 con 0x7fd93402b020 2024-03-29T11:42:53.395 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.389+0000 7fd941f9d640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:53.395 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.389+0000 7fd941f9d640 10 monclient(hunting): authenticate will time out at 1332.677522s 2024-03-29T11:42:53.397 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.392+0000 7fd93affd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fd9340c1d60 0x7fd9340bab50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:53674/0 (socket says 172.21.15.72:53674) 2024-03-29T11:42:53.397 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.392+0000 7fd93affd640 1 -- 172.21.15.72:0/3538916482 learned_addr learned my addr 172.21.15.72:0/3538916482 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:42:53.397 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.392+0000 7fd927fff640 10 client.?.objecter ms_handle_connect 0x7fd9340c1d60 2024-03-29T11:42:53.397 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.392+0000 7fd927fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:53.398 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.392+0000 7fd927fff640 1 -- 172.21.15.72:0/3538916482 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 224344514 0 0) 0x7fd934030650 con 0x7fd9340c1d60 2024-03-29T11:42:53.398 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.392+0000 7fd927fff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:53.398 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.392+0000 7fd927fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:53.398 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.392+0000 7fd927fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:53.398 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.392+0000 7fd927fff640 10 monclient(hunting): my global_id is 4512 2024-03-29T11:42:53.398 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.392+0000 7fd927fff640 1 -- 172.21.15.72:0/3538916482 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd920004eb0 con 0x7fd9340c1d60 2024-03-29T11:42:53.398 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.393+0000 7fd927fff640 1 -- 172.21.15.72:0/3538916482 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4250154977 0 0) 0x7fd920004eb0 con 0x7fd9340c1d60 2024-03-29T11:42:53.398 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.393+0000 7fd927fff640 1 -- 172.21.15.72:0/3538916482 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd934030650 con 0x7fd9340c1d60 2024-03-29T11:42:53.398 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.393+0000 7fd927fff640 1 -- 172.21.15.72:0/3538916482 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fd930001bc0 con 0x7fd9340c1d60 2024-03-29T11:42:53.398 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.393+0000 7fd927fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:53.398 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.393+0000 7fd927fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:53.398 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.393+0000 7fd927fff640 10 monclient(hunting): dump: 2024-03-29T11:42:53.398 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:42:53.398 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:53.399 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:53.399 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:53.399 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:53.399 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:53.399 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:42:53.399 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:42:53.399 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:42:53.399 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:53.399 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.393+0000 7fd927fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:53.399 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.393+0000 7fd927fff640 1 -- 172.21.15.72:0/3538916482 <== mon.0 v1:172.21.15.72:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3084289457 0 0) 0x7fd934030650 con 0x7fd9340c1d60 2024-03-29T11:42:53.399 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.393+0000 7fd927fff640 1 -- 172.21.15.72:0/3538916482 >> v1:172.21.15.72:6790/0 conn(0x7fd93402b020 legacy=0x7fd934164050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:42:53.401 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.395+0000 7fd927fff640 1 -- 172.21.15.72:0/3538916482 >> v1:172.21.15.89:6789/0 conn(0x7fd9340c4c20 legacy=0x7fd9340bb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:53.401 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.395+0000 7fd927fff640 10 monclient: _finish_hunting 0 2024-03-29T11:42:53.401 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.395+0000 7fd927fff640 1 monclient: found mon.a 2024-03-29T11:42:53.401 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.395+0000 7fd927fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:53.401 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.395+0000 7fd927fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:42:53.401 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.395+0000 7fd927fff640 1 -- 172.21.15.72:0/3538916482 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd934031980 con 0x7fd9340c1d60 2024-03-29T11:42:53.401 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.395+0000 7fd927fff640 10 monclient: _finish_auth 0 2024-03-29T11:42:53.401 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.395+0000 7fd927fff640 10 monclient: _check_auth_tickets 2024-03-29T11:42:53.401 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.395+0000 7fd927fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:53.401 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.395+0000 7fd927fff640 10 client.?.objecter ms_handle_connect 0x7fd9340c4c20 2024-03-29T11:42:53.402 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.395+0000 7fd927fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:53.402 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.395+0000 7fd941f9d640 5 monclient: authenticate success, global_id 4512 2024-03-29T11:42:53.402 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.395+0000 7fd941f9d640 10 monclient: _renew_subs 2024-03-29T11:42:53.402 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.395+0000 7fd941f9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:42:53.402 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.395+0000 7fd941f9d640 1 -- 172.21.15.72:0/3538916482 --> v1:172.21.15.72:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd93402f600 con 0x7fd9340c1d60 2024-03-29T11:42:53.402 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.395+0000 7fd941f9d640 10 client.4512.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:53.402 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.395+0000 7fd941f9d640 10 monclient: _renew_subs 2024-03-29T11:42:53.402 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.395+0000 7fd941f9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:42:53.402 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.395+0000 7fd941f9d640 1 -- 172.21.15.72:0/3538916482 --> v1:172.21.15.72:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd934030130 con 0x7fd9340c1d60 2024-03-29T11:42:53.402 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.395+0000 7fd941f9d640 1 librados: init done 2024-03-29T11:42:53.402 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.396+0000 7fd927fff640 1 -- 172.21.15.72:0/3538916482 <== mon.0 v1:172.21.15.72:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd930004bf0 con 0x7fd9340c1d60 2024-03-29T11:42:53.402 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.396+0000 7fd927fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:53.402 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.396+0000 7fd927fff640 1 -- 172.21.15.72:0/3538916482 <== mon.0 v1:172.21.15.72:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fd930001d20 con 0x7fd9340c1d60 2024-03-29T11:42:53.402 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.396+0000 7fd927fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:53.403 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.396+0000 7fd927fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:53.403 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.396+0000 7fd927fff640 10 monclient: dump: 2024-03-29T11:42:53.403 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:42:53.403 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:42:53.403 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:53.403 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:42:53.403 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:53.403 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:42:53.403 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:42:53.403 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:42:53.403 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:42:53.403 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:42:53.403 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.396+0000 7fd9397fa640 4 set_mon_vals no callback set 2024-03-29T11:42:53.403 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.398+0000 7fd927fff640 1 -- 172.21.15.72:0/3538916482 <== mon.0 v1:172.21.15.72:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7fd93001ad70 con 0x7fd9340c1d60 2024-03-29T11:42:53.405 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.398+0000 7fd941f9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:53.405 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.398+0000 7fd941f9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:53.405 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.398+0000 7fd941f9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:42:53.405 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.398+0000 7fd941f9d640 1 -- 172.21.15.72:0/3538916482 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd91c005080 con 0x7fd9340c1d60 2024-03-29T11:42:53.405 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.398+0000 7fd927fff640 1 -- 172.21.15.72:0/3538916482 <== mon.0 v1:172.21.15.72:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7fd930084d30 con 0x7fd9340c1d60 2024-03-29T11:42:53.405 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.398+0000 7fd927fff640 10 client.4512.objecter ms_dispatch 0x7fd93c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:53.405 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.398+0000 7fd927fff640 3 client.4512.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:53.405 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.398+0000 7fd927fff640 3 client.4512.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:53.405 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.399+0000 7fd927fff640 20 client.4512.objecter dump_active .. 0 homeless 2024-03-29T11:42:53.405 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.399+0000 7fd93b7fe640 10 monclient: get_auth_request con 0x7fd92006db40 auth_method 0 2024-03-29T11:42:53.405 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.400+0000 7fd927fff640 10 client.4512.objecter ms_handle_connect 0x7fd92006db40 2024-03-29T11:42:53.411 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.407+0000 7fd927fff640 1 -- 172.21.15.72:0/3538916482 <== mon.0 v1:172.21.15.72:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd930050440 con 0x7fd9340c1d60 2024-03-29T11:42:53.411 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.407+0000 7fd927fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:53.411 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.407+0000 7fd927fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:53.751 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.747+0000 7fd941f9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:53.751 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.747+0000 7fd941f9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:53.751 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.747+0000 7fd941f9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:42:53.751 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.747+0000 7fd941f9d640 1 -- 172.21.15.72:0/3538916482 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd91c005330 con 0x7fd9340c1d60 2024-03-29T11:42:53.752 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.749+0000 7fd927fff640 1 -- 172.21.15.72:0/3538916482 <== mon.0 v1:172.21.15.72:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7fd930052540 con 0x7fd9340c1d60 2024-03-29T11:42:53.752 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.749+0000 7fd927fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:53.752 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.749+0000 7fd927fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:53.753 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:42:53.753 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:53.755 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.752+0000 7fd941f9d640 10 librados: watch_flush enter 2024-03-29T11:42:53.757 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.754+0000 7fd941f9d640 10 librados: watch_flush exit 2024-03-29T11:42:53.757 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.754+0000 7fd941f9d640 20 client.4512.objecter shutdown clearing up homeless session... 2024-03-29T11:42:53.757 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.754+0000 7fd941f9d640 10 client.4512.objecter successfully canceled tick 2024-03-29T11:42:53.757 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.754+0000 7fd941f9d640 1 -- 172.21.15.72:0/3538916482 >> v1:172.21.15.72:6816/1578866102 conn(0x7fd92006db40 legacy=0x7fd920070000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:53.758 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.754+0000 7fd941f9d640 10 monclient: shutdown 2024-03-29T11:42:53.758 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.754+0000 7fd941f9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:53.758 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.754+0000 7fd941f9d640 1 -- 172.21.15.72:0/3538916482 >> v1:172.21.15.72:6789/0 conn(0x7fd9340c1d60 legacy=0x7fd9340bab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:53.758 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.754+0000 7fd941f9d640 1 -- 172.21.15.72:0/3538916482 shutdown_connections 2024-03-29T11:42:53.758 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.754+0000 7fd941f9d640 1 -- 172.21.15.72:0/3538916482 >> 172.21.15.72:0/3538916482 conn(0x7fd93403a5b0 msgr2=0x7fd934036800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:53.759 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.756+0000 7fd941f9d640 1 -- 172.21.15.72:0/3538916482 shutdown_connections 2024-03-29T11:42:53.759 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.756+0000 7fd941f9d640 1 -- 172.21.15.72:0/3538916482 wait complete. 2024-03-29T11:42:53.759 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:42:53.756+0000 7fd941f9d640 1 librados: shutdown 2024-03-29T11:42:53.777 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:00.584 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:00.733 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.729+0000 7f5adb80d640 10 monclient: get_monmap_and_config 2024-03-29T11:43:00.734 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.729+0000 7f5adb80d640 10 monclient: build_initial_monmap 2024-03-29T11:43:00.734 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.729+0000 7f5adb80d640 10 monclient: monmap: 2024-03-29T11:43:00.734 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:43:00.734 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:00.734 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:43:00.730363+0000 2024-03-29T11:43:00.734 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:43:00.730363+0000 2024-03-29T11:43:00.734 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:00.734 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:00.734 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:43:00.734 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:43:00.734 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:43:00.735 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:00.735 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.730+0000 7f5adb80d640 1 Processor -- start 2024-03-29T11:43:00.737 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.731+0000 7f5adb80d640 1 -- start start 2024-03-29T11:43:00.737 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.731+0000 7f5adb80d640 10 monclient: init 2024-03-29T11:43:00.737 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.732+0000 7f5adb80d640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:00.737 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.732+0000 7f5adb80d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:43:00.737 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.732+0000 7f5adb80d640 10 monclient(hunting): picked mon.noname-a con 0x7f5ad40a9ac0 addr v1:172.21.15.72:6789/0 2024-03-29T11:43:00.737 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.732+0000 7f5adb80d640 10 monclient(hunting): picked mon.noname-c con 0x7f5ad413ee90 addr v1:172.21.15.89:6789/0 2024-03-29T11:43:00.737 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.732+0000 7f5adb80d640 10 monclient(hunting): picked mon.noname-b con 0x7f5ad40af900 addr v1:172.21.15.72:6790/0 2024-03-29T11:43:00.737 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.732+0000 7f5adb80d640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ad40afd10 con 0x7f5ad40a9ac0 2024-03-29T11:43:00.737 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.732+0000 7f5adb80d640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ad40af6f0 con 0x7f5ad413ee90 2024-03-29T11:43:00.737 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.732+0000 7f5adb80d640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ad40aac80 con 0x7f5ad40af900 2024-03-29T11:43:00.738 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.732+0000 7f5adb80d640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:00.738 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.732+0000 7f5adb80d640 10 monclient(hunting): authenticate will time out at 1340.020109s 2024-03-29T11:43:00.738 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.733+0000 7f5ad9daa640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f5ad40af900 0x7f5ad4149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:35054/0 (socket says 172.21.15.72:35054) 2024-03-29T11:43:00.738 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.733+0000 7f5ad9daa640 1 -- 172.21.15.72:0/61947170 learned_addr learned my addr 172.21.15.72:0/61947170 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:43:00.738 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.734+0000 7f5acbfff640 1 -- 172.21.15.72:0/61947170 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3411146334 0 0) 0x7f5ad40aac80 con 0x7f5ad40af900 2024-03-29T11:43:00.738 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.734+0000 7f5acbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:43:00.738 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.734+0000 7f5acbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:00.738 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.734+0000 7f5acbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:00.738 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.734+0000 7f5acbfff640 10 monclient(hunting): my global_id is 4523 2024-03-29T11:43:00.738 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.734+0000 7f5acbfff640 1 -- 172.21.15.72:0/61947170 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ac0003520 con 0x7f5ad40af900 2024-03-29T11:43:00.739 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.734+0000 7f5acbfff640 1 -- 172.21.15.72:0/61947170 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2115975819 0 0) 0x7f5ac0003520 con 0x7f5ad40af900 2024-03-29T11:43:00.739 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.734+0000 7f5acbfff640 1 -- 172.21.15.72:0/61947170 >> v1:172.21.15.89:6789/0 conn(0x7f5ad413ee90 legacy=0x7f5ad413b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:43:00.739 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.734+0000 7f5acbfff640 1 -- 172.21.15.72:0/61947170 >> v1:172.21.15.72:6789/0 conn(0x7f5ad40a9ac0 legacy=0x7f5ad40adfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:43:00.739 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.734+0000 7f5acbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:43:00.739 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.734+0000 7f5acbfff640 1 monclient: found mon.noname-b 2024-03-29T11:43:00.739 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.734+0000 7f5acbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:00.740 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.734+0000 7f5acbfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.72:6790/0 2024-03-29T11:43:00.740 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.734+0000 7f5acbfff640 1 -- 172.21.15.72:0/61947170 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad40aaf60 con 0x7f5ad40af900 2024-03-29T11:43:00.740 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.734+0000 7f5acbfff640 10 monclient: _finish_auth 0 2024-03-29T11:43:00.740 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.734+0000 7f5acbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:43:00.740 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.734+0000 7f5acbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:00.740 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.734+0000 7f5acbfff640 1 -- 172.21.15.72:0/61947170 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f5ac4001690 con 0x7f5ad40af900 2024-03-29T11:43:00.740 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.734+0000 7f5acbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:00.740 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.734+0000 7f5acbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:00.740 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.734+0000 7f5acbfff640 10 monclient: dump: 2024-03-29T11:43:00.740 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:43:00.740 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:00.740 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:00.740 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:00.740 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:00.740 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:00.740 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:43:00.741 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:43:00.741 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:43:00.741 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:00.741 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.734+0000 7f5acbfff640 1 -- 172.21.15.72:0/61947170 <== mon.2 v1:172.21.15.72:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ac4001860 con 0x7f5ad40af900 2024-03-29T11:43:00.741 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.734+0000 7f5acbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:00.741 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.734+0000 7f5acbfff640 1 -- 172.21.15.72:0/61947170 <== mon.2 v1:172.21.15.72:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f5ac4001cb0 con 0x7f5ad40af900 2024-03-29T11:43:00.741 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.734+0000 7f5acbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:00.741 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.734+0000 7f5acbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:00.741 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.734+0000 7f5acbfff640 10 monclient: dump: 2024-03-29T11:43:00.741 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:43:00.741 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:00.741 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:00.741 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:00.741 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:00.741 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:00.742 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:43:00.742 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:43:00.742 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:43:00.742 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:00.742 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.734+0000 7f5adb80d640 5 monclient: authenticate success, global_id 4523 2024-03-29T11:43:00.742 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.734+0000 7f5adb80d640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:00.742 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.734+0000 7f5adb80d640 4 set_mon_vals no callback set 2024-03-29T11:43:00.742 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.735+0000 7f5adb80d640 10 monclient: shutdown 2024-03-29T11:43:00.742 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.735+0000 7f5adb80d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:00.742 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.735+0000 7f5adb80d640 1 -- 172.21.15.72:0/61947170 >> v1:172.21.15.72:6790/0 conn(0x7f5ad40af900 legacy=0x7f5ad4149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:00.742 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.735+0000 7f5adb80d640 1 -- 172.21.15.72:0/61947170 shutdown_connections 2024-03-29T11:43:00.742 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.735+0000 7f5adb80d640 1 -- 172.21.15.72:0/61947170 >> 172.21.15.72:0/61947170 conn(0x7f5ad40b35e0 msgr2=0x7f5ad40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:00.744 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.736+0000 7f5adb80d640 1 -- 172.21.15.72:0/61947170 shutdown_connections 2024-03-29T11:43:00.744 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.737+0000 7f5adb80d640 1 -- 172.21.15.72:0/61947170 wait complete. 2024-03-29T11:43:00.744 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.737+0000 7f5adb80d640 10 monclient: build_initial_monmap 2024-03-29T11:43:00.744 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.737+0000 7f5adb80d640 10 monclient: monmap: 2024-03-29T11:43:00.744 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:43:00.744 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:00.744 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:43:00.744 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:43:00.744 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:00.744 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:00.744 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:43:00.744 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:43:00.745 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:43:00.745 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:00.745 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.738+0000 7f5adb80d640 1 librados: starting msgr at 2024-03-29T11:43:00.745 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.738+0000 7f5adb80d640 1 librados: starting objecter 2024-03-29T11:43:00.745 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.738+0000 7f5adb80d640 1 Processor -- start 2024-03-29T11:43:00.745 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.738+0000 7f5adb80d640 1 -- start start 2024-03-29T11:43:00.745 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.738+0000 7f5adb80d640 1 librados: setting wanted keys 2024-03-29T11:43:00.745 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.738+0000 7f5adb80d640 1 librados: calling monclient init 2024-03-29T11:43:00.745 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.738+0000 7f5adb80d640 10 monclient: init 2024-03-29T11:43:00.745 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.739+0000 7f5adb80d640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:00.745 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.739+0000 7f5adb80d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:43:00.746 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.739+0000 7f5adb80d640 10 monclient(hunting): picked mon.noname-b con 0x7f5ad40a9ac0 addr v1:172.21.15.89:6789/0 2024-03-29T11:43:00.746 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.739+0000 7f5adb80d640 10 monclient(hunting): picked mon.noname-c con 0x7f5ad40af900 addr v1:172.21.15.72:6790/0 2024-03-29T11:43:00.746 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.739+0000 7f5adb80d640 10 monclient(hunting): picked mon.noname-a con 0x7f5ad413ee90 addr v1:172.21.15.72:6789/0 2024-03-29T11:43:00.746 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.739+0000 7f5adb80d640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ad40717e0 con 0x7f5ad413ee90 2024-03-29T11:43:00.746 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.739+0000 7f5adb80d640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ad40704f0 con 0x7f5ad40a9ac0 2024-03-29T11:43:00.746 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.739+0000 7f5adb80d640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ad4071430 con 0x7f5ad40af900 2024-03-29T11:43:00.746 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.739+0000 7f5adb80d640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:00.746 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.739+0000 7f5adb80d640 10 monclient(hunting): authenticate will time out at 1340.027280s 2024-03-29T11:43:00.746 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.741+0000 7f5ad8da8640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f5ad40af900 0x7f5ad4139bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:37186/0 (socket says 172.21.15.72:37186) 2024-03-29T11:43:00.746 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.741+0000 7f5ad8da8640 1 -- 172.21.15.72:0/3894007544 learned_addr learned my addr 172.21.15.72:0/3894007544 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:43:00.747 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.741+0000 7f5ac9ffb640 10 client.?.objecter ms_handle_connect 0x7f5ad40af900 2024-03-29T11:43:00.747 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.741+0000 7f5ac9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:00.747 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.742+0000 7f5ac9ffb640 1 -- 172.21.15.72:0/3894007544 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3186756752 0 0) 0x7f5ad4071430 con 0x7f5ad40af900 2024-03-29T11:43:00.747 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.742+0000 7f5ac9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:43:00.747 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.742+0000 7f5ac9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:00.747 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.742+0000 7f5ac9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:00.747 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.742+0000 7f5ac9ffb640 10 monclient(hunting): my global_id is 4526 2024-03-29T11:43:00.747 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.742+0000 7f5ac9ffb640 1 -- 172.21.15.72:0/3894007544 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ab8004eb0 con 0x7f5ad40af900 2024-03-29T11:43:00.747 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.743+0000 7f5ac9ffb640 1 -- 172.21.15.72:0/3894007544 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2564806777 0 0) 0x7f5ab8004eb0 con 0x7f5ad40af900 2024-03-29T11:43:00.747 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.743+0000 7f5ac9ffb640 1 -- 172.21.15.72:0/3894007544 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ad4071430 con 0x7f5ad40af900 2024-03-29T11:43:00.747 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.743+0000 7f5ac9ffb640 1 -- 172.21.15.72:0/3894007544 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f5ad0001870 con 0x7f5ad40af900 2024-03-29T11:43:00.747 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.743+0000 7f5ac9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:00.747 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.743+0000 7f5ac9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:00.747 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.743+0000 7f5ac9ffb640 10 monclient(hunting): dump: 2024-03-29T11:43:00.747 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:43:00.748 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:00.748 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:00.748 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:00.748 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:00.748 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:00.748 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:43:00.748 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:43:00.748 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:43:00.748 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:00.748 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.743+0000 7f5ac9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:00.749 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.744+0000 7f5ac9ffb640 10 client.?.objecter ms_handle_connect 0x7f5ad40a9ac0 2024-03-29T11:43:00.749 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.744+0000 7f5ac9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:00.749 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.745+0000 7f5ac9ffb640 1 -- 172.21.15.72:0/3894007544 <== mon.2 v1:172.21.15.72:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 153677090 0 0) 0x7f5ad4071430 con 0x7f5ad40af900 2024-03-29T11:43:00.749 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.745+0000 7f5ac9ffb640 1 -- 172.21.15.72:0/3894007544 >> v1:172.21.15.89:6789/0 conn(0x7f5ad40a9ac0 legacy=0x7f5ad4070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:00.749 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.745+0000 7f5ac9ffb640 1 -- 172.21.15.72:0/3894007544 >> v1:172.21.15.72:6789/0 conn(0x7f5ad413ee90 legacy=0x7f5ad41da700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:00.749 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.745+0000 7f5ac9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:43:00.749 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.745+0000 7f5ac9ffb640 1 monclient: found mon.c 2024-03-29T11:43:00.749 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.745+0000 7f5ac9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:00.749 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.745+0000 7f5ac9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:43:00.749 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.745+0000 7f5ac9ffb640 1 -- 172.21.15.72:0/3894007544 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad4141da0 con 0x7f5ad40af900 2024-03-29T11:43:00.750 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.745+0000 7f5ac9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:43:00.750 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.745+0000 7f5ac9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:43:00.750 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.745+0000 7f5ac9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:00.750 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.745+0000 7f5ac9ffb640 1 -- 172.21.15.72:0/3894007544 <== mon.2 v1:172.21.15.72:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ad0004920 con 0x7f5ad40af900 2024-03-29T11:43:00.750 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.745+0000 7f5ac9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:00.750 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.745+0000 7f5ac9ffb640 1 -- 172.21.15.72:0/3894007544 <== mon.2 v1:172.21.15.72:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f5ad0001870 con 0x7f5ad40af900 2024-03-29T11:43:00.750 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.745+0000 7f5ac9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:00.750 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.745+0000 7f5ac9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:00.750 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.745+0000 7f5ac9ffb640 10 monclient: dump: 2024-03-29T11:43:00.750 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:43:00.750 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:00.750 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:00.750 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:00.751 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:00.751 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:00.751 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:43:00.751 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:43:00.751 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:43:00.751 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:00.751 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.745+0000 7f5adb80d640 5 monclient: authenticate success, global_id 4526 2024-03-29T11:43:00.751 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.745+0000 7f5adb80d640 10 monclient: _renew_subs 2024-03-29T11:43:00.751 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.745+0000 7f5adb80d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:43:00.751 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.745+0000 7f5adb80d640 1 -- 172.21.15.72:0/3894007544 --> v1:172.21.15.72:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ad4142010 con 0x7f5ad40af900 2024-03-29T11:43:00.751 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.745+0000 7f5adb80d640 10 client.4526.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:00.751 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.745+0000 7f5adb80d640 10 monclient: _renew_subs 2024-03-29T11:43:00.751 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.745+0000 7f5adb80d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:43:00.751 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.745+0000 7f5adb80d640 1 -- 172.21.15.72:0/3894007544 --> v1:172.21.15.72:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ad413a4b0 con 0x7f5ad40af900 2024-03-29T11:43:00.751 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.745+0000 7f5adb80d640 1 librados: init done 2024-03-29T11:43:00.753 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.747+0000 7f5acb7fe640 4 set_mon_vals no callback set 2024-03-29T11:43:00.753 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.747+0000 7f5adb80d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:00.753 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.748+0000 7f5adb80d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:00.753 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.748+0000 7f5adb80d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:43:00.753 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.748+0000 7f5adb80d640 1 -- 172.21.15.72:0/3894007544 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ab4005080 con 0x7f5ad40af900 2024-03-29T11:43:00.753 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.748+0000 7f5ac9ffb640 1 -- 172.21.15.72:0/3894007544 <== mon.2 v1:172.21.15.72:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f5ad001a900 con 0x7f5ad40af900 2024-03-29T11:43:00.753 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.749+0000 7f5ac9ffb640 1 -- 172.21.15.72:0/3894007544 <== mon.2 v1:172.21.15.72:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f5ad00544f0 con 0x7f5ad40af900 2024-03-29T11:43:00.753 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.749+0000 7f5ac9ffb640 10 client.4526.objecter ms_dispatch 0x7f5ad4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:00.753 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.749+0000 7f5ac9ffb640 3 client.4526.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:00.753 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.749+0000 7f5ac9ffb640 3 client.4526.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:00.754 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.749+0000 7f5ac9ffb640 20 client.4526.objecter dump_active .. 0 homeless 2024-03-29T11:43:00.754 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.749+0000 7f5ad95a9640 10 monclient: get_auth_request con 0x7f5ab8076840 auth_method 0 2024-03-29T11:43:00.754 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.751+0000 7f5ac9ffb640 10 client.4526.objecter ms_handle_connect 0x7f5ab8076840 2024-03-29T11:43:00.761 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.758+0000 7f5ac9ffb640 1 -- 172.21.15.72:0/3894007544 <== mon.2 v1:172.21.15.72:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5ad0050790 con 0x7f5ad40af900 2024-03-29T11:43:00.761 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.758+0000 7f5ac9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:00.762 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:00.758+0000 7f5ac9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:01.088 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:01.083+0000 7f5adb80d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:01.088 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:01.083+0000 7f5adb80d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:01.088 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:01.083+0000 7f5adb80d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:43:01.088 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:01.083+0000 7f5adb80d640 1 -- 172.21.15.72:0/3894007544 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ab4005330 con 0x7f5ad40af900 2024-03-29T11:43:01.088 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:01.085+0000 7f5ac9ffb640 1 -- 172.21.15.72:0/3894007544 <== mon.2 v1:172.21.15.72:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f5ad0050790 con 0x7f5ad40af900 2024-03-29T11:43:01.088 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:01.085+0000 7f5ac9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:01.089 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:01.085+0000 7f5ac9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:01.089 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:43:01.089 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:01.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:01.088+0000 7f5adb80d640 10 librados: watch_flush enter 2024-03-29T11:43:01.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:01.088+0000 7f5adb80d640 10 librados: watch_flush exit 2024-03-29T11:43:01.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:01.088+0000 7f5adb80d640 20 client.4526.objecter shutdown clearing up homeless session... 2024-03-29T11:43:01.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:01.088+0000 7f5adb80d640 10 client.4526.objecter successfully canceled tick 2024-03-29T11:43:01.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:01.088+0000 7f5adb80d640 1 -- 172.21.15.72:0/3894007544 >> v1:172.21.15.72:6816/1578866102 conn(0x7f5ab8076840 legacy=0x7f5ab8078d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:01.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:01.088+0000 7f5adb80d640 10 monclient: shutdown 2024-03-29T11:43:01.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:01.089+0000 7f5adb80d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:01.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:01.089+0000 7f5adb80d640 1 -- 172.21.15.72:0/3894007544 >> v1:172.21.15.72:6790/0 conn(0x7f5ad40af900 legacy=0x7f5ad4139bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:01.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:01.089+0000 7f5adb80d640 1 -- 172.21.15.72:0/3894007544 shutdown_connections 2024-03-29T11:43:01.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:01.089+0000 7f5adb80d640 1 -- 172.21.15.72:0/3894007544 >> 172.21.15.72:0/3894007544 conn(0x7f5ad40b35e0 msgr2=0x7f5ad41ccf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:01.093 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:01.089+0000 7f5adb80d640 1 -- 172.21.15.72:0/3894007544 shutdown_connections 2024-03-29T11:43:01.093 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:01.089+0000 7f5adb80d640 1 -- 172.21.15.72:0/3894007544 wait complete. 2024-03-29T11:43:01.093 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:01.090+0000 7f5adb80d640 1 librados: shutdown 2024-03-29T11:43:01.112 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:07.919 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:08.062 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.059+0000 7fec4e332640 10 monclient: get_monmap_and_config 2024-03-29T11:43:08.063 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.059+0000 7fec4e332640 10 monclient: build_initial_monmap 2024-03-29T11:43:08.063 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.059+0000 7fec4e332640 10 monclient: monmap: 2024-03-29T11:43:08.063 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:43:08.063 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:08.063 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:43:08.060331+0000 2024-03-29T11:43:08.063 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:43:08.060331+0000 2024-03-29T11:43:08.063 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:08.064 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:08.064 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:43:08.064 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:43:08.064 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:43:08.064 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:08.065 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.060+0000 7fec4e332640 1 Processor -- start 2024-03-29T11:43:08.068 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.060+0000 7fec4e332640 1 -- start start 2024-03-29T11:43:08.068 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.060+0000 7fec4e332640 10 monclient: init 2024-03-29T11:43:08.068 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.061+0000 7fec4e332640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:08.068 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.061+0000 7fec4e332640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:43:08.068 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.061+0000 7fec4e332640 10 monclient(hunting): picked mon.noname-b con 0x7fec481457f0 addr v1:172.21.15.72:6790/0 2024-03-29T11:43:08.068 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.061+0000 7fec4e332640 10 monclient(hunting): picked mon.noname-a con 0x7fec4813f650 addr v1:172.21.15.72:6789/0 2024-03-29T11:43:08.068 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.061+0000 7fec4e332640 10 monclient(hunting): picked mon.noname-c con 0x7fec480a9d70 addr v1:172.21.15.89:6789/0 2024-03-29T11:43:08.068 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.061+0000 7fec4e332640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec480aaef0 con 0x7fec4813f650 2024-03-29T11:43:08.069 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.061+0000 7fec4e332640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec48145bd0 con 0x7fec480a9d70 2024-03-29T11:43:08.069 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.061+0000 7fec4e332640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec48145eb0 con 0x7fec481457f0 2024-03-29T11:43:08.069 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.061+0000 7fec4e332640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:08.069 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.061+0000 7fec4e332640 10 monclient(hunting): authenticate will time out at 1347.349319s 2024-03-29T11:43:08.069 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.063+0000 7fec4c8cf640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fec480a9d70 0x7fec4814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:56493/0 (socket says 172.21.15.72:56493) 2024-03-29T11:43:08.069 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.063+0000 7fec4c8cf640 1 -- 172.21.15.72:0/3933290493 learned_addr learned my addr 172.21.15.72:0/3933290493 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:43:08.069 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.063+0000 7fec4c8cf640 1 -- 172.21.15.72:0/3933290493 >> v1:172.21.15.89:6789/0 conn(0x7fec480a9d70 legacy=0x7fec4814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-29T11:43:08.069 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.063+0000 7fec4c8cf640 1 -- 172.21.15.72:0/3933290493 >> v1:172.21.15.89:6789/0 conn(0x7fec480a9d70 legacy=0x7fec4814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:43:08.069 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.063+0000 7fec4c8cf640 1 --1- 172.21.15.72:0/3933290493 >> v1:172.21.15.89:6789/0 conn(0x7fec480a9d70 0x7fec4814d8c0 :-1 s=OPENED pgs=990646 cs=1 l=1).handle_message read tag failed 2024-03-29T11:43:08.069 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.063+0000 7fec4c8cf640 1 --1- 172.21.15.72:0/3933290493 >> v1:172.21.15.89:6789/0 conn(0x7fec480a9d70 0x7fec4814d8c0 :-1 s=OPENED pgs=990646 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:43:08.069 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.064+0000 7fec46ffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.89:6789/0 2024-03-29T11:43:08.069 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.064+0000 7fec46ffd640 1 -- 172.21.15.72:0/3933290493 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1812196859 0 0) 0x7fec48145eb0 con 0x7fec481457f0 2024-03-29T11:43:08.070 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.064+0000 7fec46ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:08.070 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.064+0000 7fec46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:08.070 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.064+0000 7fec46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:08.070 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.064+0000 7fec46ffd640 10 monclient(hunting): my global_id is 4529 2024-03-29T11:43:08.070 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.064+0000 7fec46ffd640 1 -- 172.21.15.72:0/3933290493 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec30003520 con 0x7fec481457f0 2024-03-29T11:43:08.070 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.064+0000 7fec46ffd640 1 -- 172.21.15.72:0/3933290493 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3044688403 0 0) 0x7fec30003520 con 0x7fec481457f0 2024-03-29T11:43:08.070 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.064+0000 7fec46ffd640 1 -- 172.21.15.72:0/3933290493 >> v1:172.21.15.89:6789/0 conn(0x7fec480a9d70 legacy=0x7fec4814d8c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:43:08.070 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.064+0000 7fec46ffd640 1 -- 172.21.15.72:0/3933290493 >> v1:172.21.15.72:6789/0 conn(0x7fec4813f650 legacy=0x7fec480b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:43:08.070 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.065+0000 7fec46ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:43:08.073 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.065+0000 7fec46ffd640 1 monclient: found mon.noname-b 2024-03-29T11:43:08.073 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.065+0000 7fec46ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:08.073 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.065+0000 7fec46ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.72:6790/0 2024-03-29T11:43:08.073 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.065+0000 7fec46ffd640 1 -- 172.21.15.72:0/3933290493 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec48146340 con 0x7fec481457f0 2024-03-29T11:43:08.073 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.065+0000 7fec46ffd640 10 monclient: _finish_auth 0 2024-03-29T11:43:08.073 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.065+0000 7fec46ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:43:08.073 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.065+0000 7fec46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:08.073 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.065+0000 7fec46ffd640 1 -- 172.21.15.72:0/3933290493 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fec38001690 con 0x7fec481457f0 2024-03-29T11:43:08.073 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.065+0000 7fec46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:08.073 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.065+0000 7fec46ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:08.074 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.065+0000 7fec46ffd640 10 monclient: dump: 2024-03-29T11:43:08.074 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:43:08.074 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:08.074 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:08.074 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:08.074 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:08.074 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:08.074 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:43:08.074 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:43:08.074 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:43:08.074 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:08.074 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.065+0000 7fec4e332640 5 monclient: authenticate success, global_id 4529 2024-03-29T11:43:08.074 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.065+0000 7fec4e332640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:08.075 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.067+0000 7fec46ffd640 1 -- 172.21.15.72:0/3933290493 <== mon.2 v1:172.21.15.72:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec380017f0 con 0x7fec481457f0 2024-03-29T11:43:08.075 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.067+0000 7fec46ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:08.075 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.067+0000 7fec46ffd640 1 -- 172.21.15.72:0/3933290493 <== mon.2 v1:172.21.15.72:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fec38001c40 con 0x7fec481457f0 2024-03-29T11:43:08.075 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.067+0000 7fec46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:08.075 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.067+0000 7fec46ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:08.075 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.067+0000 7fec46ffd640 10 monclient: dump: 2024-03-29T11:43:08.075 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:43:08.075 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:08.075 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:08.075 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:08.075 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:08.075 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:08.075 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:43:08.075 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:43:08.076 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:43:08.076 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:08.076 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.068+0000 7fec4e332640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:08.076 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.068+0000 7fec4e332640 4 set_mon_vals no callback set 2024-03-29T11:43:08.077 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.070+0000 7fec4e332640 10 monclient: shutdown 2024-03-29T11:43:08.077 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.070+0000 7fec4e332640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:08.078 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.070+0000 7fec4e332640 1 -- 172.21.15.72:0/3933290493 >> v1:172.21.15.72:6790/0 conn(0x7fec481457f0 legacy=0x7fec4813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:08.078 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.072+0000 7fec4e332640 1 -- 172.21.15.72:0/3933290493 shutdown_connections 2024-03-29T11:43:08.078 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.072+0000 7fec4e332640 1 -- 172.21.15.72:0/3933290493 >> 172.21.15.72:0/3933290493 conn(0x7fec48066a00 msgr2=0x7fec48066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:08.078 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.074+0000 7fec4e332640 1 -- 172.21.15.72:0/3933290493 shutdown_connections 2024-03-29T11:43:08.078 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.074+0000 7fec4e332640 1 -- 172.21.15.72:0/3933290493 wait complete. 2024-03-29T11:43:08.081 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.075+0000 7fec4e332640 10 monclient: build_initial_monmap 2024-03-29T11:43:08.081 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.075+0000 7fec4e332640 10 monclient: monmap: 2024-03-29T11:43:08.081 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:43:08.081 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:08.082 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:43:08.082 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:43:08.082 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:08.082 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:08.082 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:43:08.083 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:43:08.083 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:43:08.083 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:08.083 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.075+0000 7fec4e332640 1 librados: starting msgr at 2024-03-29T11:43:08.083 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.075+0000 7fec4e332640 1 librados: starting objecter 2024-03-29T11:43:08.083 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.075+0000 7fec4e332640 1 Processor -- start 2024-03-29T11:43:08.083 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.075+0000 7fec4e332640 1 -- start start 2024-03-29T11:43:08.083 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.075+0000 7fec4e332640 1 librados: setting wanted keys 2024-03-29T11:43:08.084 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.075+0000 7fec4e332640 1 librados: calling monclient init 2024-03-29T11:43:08.084 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.075+0000 7fec4e332640 10 monclient: init 2024-03-29T11:43:08.084 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.076+0000 7fec4e332640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:08.084 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.076+0000 7fec4e332640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:43:08.084 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.076+0000 7fec4e332640 10 monclient(hunting): picked mon.noname-b con 0x7fec480a9d70 addr v1:172.21.15.89:6789/0 2024-03-29T11:43:08.084 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.076+0000 7fec4e332640 10 monclient(hunting): picked mon.noname-c con 0x7fec4813f650 addr v1:172.21.15.72:6790/0 2024-03-29T11:43:08.084 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.076+0000 7fec4e332640 10 monclient(hunting): picked mon.noname-a con 0x7fec481457f0 addr v1:172.21.15.72:6789/0 2024-03-29T11:43:08.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.076+0000 7fec4e332640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec48070840 con 0x7fec481457f0 2024-03-29T11:43:08.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.076+0000 7fec4e332640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec481422e0 con 0x7fec480a9d70 2024-03-29T11:43:08.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.076+0000 7fec4e332640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec48142470 con 0x7fec4813f650 2024-03-29T11:43:08.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.076+0000 7fec4e332640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:08.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.076+0000 7fec4e332640 10 monclient(hunting): authenticate will time out at 1347.364765s 2024-03-29T11:43:08.093 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.080+0000 7fec477fe640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7fec4813f650 0x7fec481db670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:42814/0 (socket says 172.21.15.72:42814) 2024-03-29T11:43:08.094 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.080+0000 7fec4c8cf640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fec481457f0 0x7fec481e6eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:59757/0 (socket says 172.21.15.72:59757) 2024-03-29T11:43:08.094 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.080+0000 7fec4c8cf640 1 -- 172.21.15.72:0/2014914201 learned_addr learned my addr 172.21.15.72:0/2014914201 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:43:08.094 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.081+0000 7fec44ff9640 10 client.?.objecter ms_handle_connect 0x7fec481457f0 2024-03-29T11:43:08.094 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.081+0000 7fec44ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:08.094 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.081+0000 7fec44ff9640 10 client.?.objecter ms_handle_connect 0x7fec4813f650 2024-03-29T11:43:08.094 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.081+0000 7fec44ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:08.094 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.082+0000 7fec44ff9640 1 -- 172.21.15.72:0/2014914201 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3045393535 0 0) 0x7fec48142470 con 0x7fec4813f650 2024-03-29T11:43:08.094 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.082+0000 7fec44ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:43:08.094 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.082+0000 7fec44ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:08.094 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.082+0000 7fec44ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:08.095 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.082+0000 7fec44ff9640 10 monclient(hunting): my global_id is 4532 2024-03-29T11:43:08.095 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.082+0000 7fec44ff9640 1 -- 172.21.15.72:0/2014914201 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec34004eb0 con 0x7fec4813f650 2024-03-29T11:43:08.095 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.083+0000 7fec44ff9640 1 -- 172.21.15.72:0/2014914201 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1135224876 0 0) 0x7fec48070840 con 0x7fec481457f0 2024-03-29T11:43:08.095 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.083+0000 7fec44ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:43:08.095 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.083+0000 7fec44ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:08.095 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.083+0000 7fec44ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:08.095 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.083+0000 7fec44ff9640 10 monclient(hunting): my global_id is 4515 2024-03-29T11:43:08.095 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.083+0000 7fec44ff9640 1 -- 172.21.15.72:0/2014914201 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec340051d0 con 0x7fec481457f0 2024-03-29T11:43:08.095 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.083+0000 7fec44ff9640 1 -- 172.21.15.72:0/2014914201 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1412471625 0 0) 0x7fec34004eb0 con 0x7fec4813f650 2024-03-29T11:43:08.096 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.083+0000 7fec44ff9640 1 -- 172.21.15.72:0/2014914201 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec48070840 con 0x7fec4813f650 2024-03-29T11:43:08.096 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.083+0000 7fec44ff9640 1 -- 172.21.15.72:0/2014914201 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fec3c001860 con 0x7fec4813f650 2024-03-29T11:43:08.096 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.083+0000 7fec44ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:08.096 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.083+0000 7fec44ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:08.096 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.083+0000 7fec44ff9640 10 monclient(hunting): dump: 2024-03-29T11:43:08.096 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:43:08.096 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:08.096 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:08.096 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:08.097 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:08.097 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:08.097 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:43:08.097 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:43:08.097 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:43:08.097 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:08.097 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.083+0000 7fec44ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:08.097 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.083+0000 7fec44ff9640 10 client.?.objecter ms_handle_connect 0x7fec480a9d70 2024-03-29T11:43:08.098 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.083+0000 7fec44ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:08.098 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.083+0000 7fec44ff9640 1 -- 172.21.15.72:0/2014914201 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 146295353 0 0) 0x7fec340051d0 con 0x7fec481457f0 2024-03-29T11:43:08.098 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.083+0000 7fec44ff9640 1 -- 172.21.15.72:0/2014914201 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec34005b60 con 0x7fec481457f0 2024-03-29T11:43:08.098 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.083+0000 7fec44ff9640 1 -- 172.21.15.72:0/2014914201 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fec40001c20 con 0x7fec481457f0 2024-03-29T11:43:08.098 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.083+0000 7fec44ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:08.098 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.083+0000 7fec44ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:08.098 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.083+0000 7fec44ff9640 10 monclient(hunting): dump: 2024-03-29T11:43:08.098 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:43:08.099 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:08.099 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:08.099 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:08.099 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:08.099 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:08.099 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:43:08.099 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:43:08.099 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:43:08.099 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:08.100 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.084+0000 7fec44ff9640 1 -- 172.21.15.72:0/2014914201 <== mon.2 v1:172.21.15.72:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1377096481 0 0) 0x7fec48070840 con 0x7fec4813f650 2024-03-29T11:43:08.100 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.084+0000 7fec44ff9640 1 -- 172.21.15.72:0/2014914201 >> v1:172.21.15.89:6789/0 conn(0x7fec480a9d70 legacy=0x7fec48135410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:08.100 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.084+0000 7fec44ff9640 1 -- 172.21.15.72:0/2014914201 >> v1:172.21.15.72:6789/0 conn(0x7fec481457f0 legacy=0x7fec481e6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:08.100 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.084+0000 7fec44ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:43:08.100 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.084+0000 7fec44ff9640 1 monclient: found mon.c 2024-03-29T11:43:08.100 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.084+0000 7fec44ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:08.100 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.084+0000 7fec44ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:43:08.100 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.084+0000 7fec44ff9640 1 -- 172.21.15.72:0/2014914201 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec48142600 con 0x7fec4813f650 2024-03-29T11:43:08.100 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.084+0000 7fec44ff9640 10 monclient: _finish_auth 0 2024-03-29T11:43:08.101 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.084+0000 7fec44ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:43:08.101 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.084+0000 7fec44ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:08.101 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.084+0000 7fec4e332640 5 monclient: authenticate success, global_id 4532 2024-03-29T11:43:08.101 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.084+0000 7fec4e332640 10 monclient: _renew_subs 2024-03-29T11:43:08.101 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.084+0000 7fec4e332640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:43:08.101 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.084+0000 7fec4e332640 1 -- 172.21.15.72:0/2014914201 --> v1:172.21.15.72:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec48070220 con 0x7fec4813f650 2024-03-29T11:43:08.101 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.084+0000 7fec4e332640 10 client.4532.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:08.101 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.084+0000 7fec4e332640 10 monclient: _renew_subs 2024-03-29T11:43:08.101 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.084+0000 7fec4e332640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:43:08.101 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.084+0000 7fec4e332640 1 -- 172.21.15.72:0/2014914201 --> v1:172.21.15.72:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fec48135b20 con 0x7fec4813f650 2024-03-29T11:43:08.101 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.084+0000 7fec4e332640 1 librados: init done 2024-03-29T11:43:08.101 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.085+0000 7fec4e332640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:08.102 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.085+0000 7fec4e332640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:08.102 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.085+0000 7fec4e332640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:43:08.102 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.085+0000 7fec4e332640 1 -- 172.21.15.72:0/2014914201 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec48070a80 con 0x7fec4813f650 2024-03-29T11:43:08.102 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.085+0000 7fec44ff9640 1 -- 172.21.15.72:0/2014914201 <== mon.2 v1:172.21.15.72:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec3c001d70 con 0x7fec4813f650 2024-03-29T11:43:08.102 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.085+0000 7fec44ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:08.102 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.085+0000 7fec467fc640 4 set_mon_vals no callback set 2024-03-29T11:43:08.102 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.086+0000 7fec44ff9640 1 -- 172.21.15.72:0/2014914201 <== mon.2 v1:172.21.15.72:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fec3c001860 con 0x7fec4813f650 2024-03-29T11:43:08.102 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.086+0000 7fec44ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:08.102 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.086+0000 7fec44ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:08.102 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.086+0000 7fec44ff9640 10 monclient: dump: 2024-03-29T11:43:08.102 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:43:08.102 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:08.102 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:08.103 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:08.103 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:08.103 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:08.103 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:43:08.103 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:43:08.103 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:43:08.103 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:08.103 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.087+0000 7fec44ff9640 1 -- 172.21.15.72:0/2014914201 <== mon.2 v1:172.21.15.72:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7fec3c003670 con 0x7fec4813f650 2024-03-29T11:43:08.103 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.088+0000 7fec44ff9640 1 -- 172.21.15.72:0/2014914201 <== mon.2 v1:172.21.15.72:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7fec3c001680 con 0x7fec4813f650 2024-03-29T11:43:08.103 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.088+0000 7fec44ff9640 10 client.4532.objecter ms_dispatch 0x7fec48000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:08.103 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.088+0000 7fec44ff9640 3 client.4532.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:08.103 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.088+0000 7fec44ff9640 3 client.4532.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:08.103 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.088+0000 7fec44ff9640 20 client.4532.objecter dump_active .. 0 homeless 2024-03-29T11:43:08.103 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.089+0000 7fec47fff640 10 monclient: get_auth_request con 0x7fec3406e4f0 auth_method 0 2024-03-29T11:43:08.103 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.089+0000 7fec44ff9640 10 client.4532.objecter ms_handle_connect 0x7fec3406e4f0 2024-03-29T11:43:08.104 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.096+0000 7fec44ff9640 1 -- 172.21.15.72:0/2014914201 <== mon.2 v1:172.21.15.72:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fec3c050460 con 0x7fec4813f650 2024-03-29T11:43:08.104 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.096+0000 7fec44ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:08.104 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.096+0000 7fec44ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:08.423 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.419+0000 7fec4e332640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:08.423 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.419+0000 7fec4e332640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:08.423 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.419+0000 7fec4e332640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:43:08.423 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.419+0000 7fec4e332640 1 -- 172.21.15.72:0/2014914201 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec48070650 con 0x7fec4813f650 2024-03-29T11:43:08.424 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.421+0000 7fec44ff9640 1 -- 172.21.15.72:0/2014914201 <== mon.2 v1:172.21.15.72:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7fec3c050600 con 0x7fec4813f650 2024-03-29T11:43:08.424 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.421+0000 7fec44ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:08.424 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.421+0000 7fec44ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:08.426 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:43:08.426 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:08.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.424+0000 7fec4e332640 10 librados: watch_flush enter 2024-03-29T11:43:08.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.424+0000 7fec4e332640 10 librados: watch_flush exit 2024-03-29T11:43:08.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.424+0000 7fec4e332640 20 client.4532.objecter shutdown clearing up homeless session... 2024-03-29T11:43:08.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.424+0000 7fec4e332640 10 client.4532.objecter successfully canceled tick 2024-03-29T11:43:08.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.424+0000 7fec4e332640 1 -- 172.21.15.72:0/2014914201 >> v1:172.21.15.72:6816/1578866102 conn(0x7fec3406e4f0 legacy=0x7fec340709b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:08.430 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.424+0000 7fec4e332640 10 monclient: shutdown 2024-03-29T11:43:08.430 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.424+0000 7fec4e332640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:08.430 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.424+0000 7fec4e332640 1 -- 172.21.15.72:0/2014914201 >> v1:172.21.15.72:6790/0 conn(0x7fec4813f650 legacy=0x7fec481db670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:08.430 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.425+0000 7fec4e332640 1 -- 172.21.15.72:0/2014914201 shutdown_connections 2024-03-29T11:43:08.430 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.425+0000 7fec4e332640 1 -- 172.21.15.72:0/2014914201 >> 172.21.15.72:0/2014914201 conn(0x7fec48066a00 msgr2=0x7fec480b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:08.430 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.425+0000 7fec4e332640 1 -- 172.21.15.72:0/2014914201 shutdown_connections 2024-03-29T11:43:08.430 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.425+0000 7fec4e332640 1 -- 172.21.15.72:0/2014914201 wait complete. 2024-03-29T11:43:08.430 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:08.426+0000 7fec4e332640 1 librados: shutdown 2024-03-29T11:43:08.444 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:15.252 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:15.404 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.401+0000 7f1e1a4f6640 10 monclient: get_monmap_and_config 2024-03-29T11:43:15.405 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.401+0000 7f1e1a4f6640 10 monclient: build_initial_monmap 2024-03-29T11:43:15.405 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.401+0000 7f1e1a4f6640 10 monclient: monmap: 2024-03-29T11:43:15.405 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:43:15.405 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:15.405 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:43:15.402453+0000 2024-03-29T11:43:15.405 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:43:15.402453+0000 2024-03-29T11:43:15.405 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:15.405 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:15.405 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:43:15.405 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:43:15.405 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:43:15.405 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:15.408 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.405+0000 7f1e1a4f6640 1 Processor -- start 2024-03-29T11:43:15.409 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.405+0000 7f1e1a4f6640 1 -- start start 2024-03-29T11:43:15.409 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.405+0000 7f1e1a4f6640 10 monclient: init 2024-03-29T11:43:15.410 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.406+0000 7f1e1a4f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:15.410 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.406+0000 7f1e1a4f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:43:15.414 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.410+0000 7f1e1a4f6640 10 monclient(hunting): picked mon.noname-c con 0x7f1e141422b0 addr v1:172.21.15.89:6789/0 2024-03-29T11:43:15.414 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.410+0000 7f1e1a4f6640 10 monclient(hunting): picked mon.noname-b con 0x7f1e14142690 addr v1:172.21.15.72:6790/0 2024-03-29T11:43:15.414 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.410+0000 7f1e1a4f6640 10 monclient(hunting): picked mon.noname-a con 0x7f1e140b17d0 addr v1:172.21.15.72:6789/0 2024-03-29T11:43:15.414 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.410+0000 7f1e1a4f6640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e14142c80 con 0x7f1e140b17d0 2024-03-29T11:43:15.414 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.410+0000 7f1e1a4f6640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e140aae40 con 0x7f1e141422b0 2024-03-29T11:43:15.414 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.410+0000 7f1e1a4f6640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e1413fbd0 con 0x7f1e14142690 2024-03-29T11:43:15.414 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.410+0000 7f1e1a4f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:15.414 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.410+0000 7f1e1a4f6640 10 monclient(hunting): authenticate will time out at 1354.698372s 2024-03-29T11:43:15.414 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.410+0000 7f1e13fff640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f1e141422b0 0x7f1e1413f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:46701/0 (socket says 172.21.15.72:46701) 2024-03-29T11:43:15.415 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.410+0000 7f1e13fff640 1 -- 172.21.15.72:0/3569833137 learned_addr learned my addr 172.21.15.72:0/3569833137 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:43:15.416 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.413+0000 7f1e12ffd640 1 -- 172.21.15.72:0/3569833137 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2978449926 0 0) 0x7f1e140aae40 con 0x7f1e141422b0 2024-03-29T11:43:15.416 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.413+0000 7f1e12ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:15.416 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.413+0000 7f1e12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:15.416 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.413+0000 7f1e12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:15.416 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.413+0000 7f1e12ffd640 10 monclient(hunting): my global_id is 4462 2024-03-29T11:43:15.416 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.413+0000 7f1e12ffd640 1 -- 172.21.15.72:0/3569833137 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dfc003520 con 0x7f1e141422b0 2024-03-29T11:43:15.417 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.413+0000 7f1e12ffd640 1 -- 172.21.15.72:0/3569833137 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1707546155 0 0) 0x7f1e14142c80 con 0x7f1e140b17d0 2024-03-29T11:43:15.417 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.413+0000 7f1e12ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:15.417 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.413+0000 7f1e12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:15.417 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.413+0000 7f1e12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:15.417 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.413+0000 7f1e12ffd640 10 monclient(hunting): my global_id is 4518 2024-03-29T11:43:15.417 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.413+0000 7f1e12ffd640 1 -- 172.21.15.72:0/3569833137 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e140aae40 con 0x7f1e140b17d0 2024-03-29T11:43:15.417 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.414+0000 7f1e12ffd640 1 -- 172.21.15.72:0/3569833137 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3205144690 0 0) 0x7f1e140aae40 con 0x7f1e140b17d0 2024-03-29T11:43:15.418 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.414+0000 7f1e12ffd640 1 -- 172.21.15.72:0/3569833137 >> v1:172.21.15.72:6790/0 conn(0x7f1e14142690 legacy=0x7f1e140b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:43:15.418 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.414+0000 7f1e12ffd640 1 -- 172.21.15.72:0/3569833137 >> v1:172.21.15.89:6789/0 conn(0x7f1e141422b0 legacy=0x7f1e1413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:15.418 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.414+0000 7f1e12ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:43:15.418 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.414+0000 7f1e12ffd640 1 monclient: found mon.noname-a 2024-03-29T11:43:15.418 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.414+0000 7f1e12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:15.418 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.414+0000 7f1e12ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.72:6789/0 2024-03-29T11:43:15.418 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.414+0000 7f1e12ffd640 1 -- 172.21.15.72:0/3569833137 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e14151940 con 0x7f1e140b17d0 2024-03-29T11:43:15.418 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.414+0000 7f1e12ffd640 10 monclient: _finish_auth 0 2024-03-29T11:43:15.418 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.414+0000 7f1e12ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:43:15.418 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.414+0000 7f1e12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:15.418 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.414+0000 7f1e12ffd640 1 -- 172.21.15.72:0/3569833137 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f1e0c0019f0 con 0x7f1e140b17d0 2024-03-29T11:43:15.418 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.414+0000 7f1e12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:15.418 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.414+0000 7f1e12ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:43:15.418 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.414+0000 7f1e12ffd640 10 monclient: dump: 2024-03-29T11:43:15.418 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:43:15.418 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:15.419 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:15.419 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:15.419 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:15.419 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:15.419 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:43:15.419 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:43:15.419 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:43:15.419 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:15.419 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.414+0000 7f1e1a4f6640 5 monclient: authenticate success, global_id 4518 2024-03-29T11:43:15.419 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.414+0000 7f1e1a4f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:15.419 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.414+0000 7f1e12ffd640 1 -- 172.21.15.72:0/3569833137 <== mon.0 v1:172.21.15.72:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1e0c001b50 con 0x7f1e140b17d0 2024-03-29T11:43:15.419 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.414+0000 7f1e12ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:15.419 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.414+0000 7f1e12ffd640 1 -- 172.21.15.72:0/3569833137 <== mon.0 v1:172.21.15.72:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f1e0c001ee0 con 0x7f1e140b17d0 2024-03-29T11:43:15.419 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.414+0000 7f1e12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:15.419 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.414+0000 7f1e12ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:15.419 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.414+0000 7f1e12ffd640 10 monclient: dump: 2024-03-29T11:43:15.420 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:43:15.420 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:15.420 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:15.420 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:15.420 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:15.420 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:15.420 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:43:15.420 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:43:15.420 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:43:15.420 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:15.420 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.414+0000 7f1e1a4f6640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:15.420 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.414+0000 7f1e1a4f6640 4 set_mon_vals no callback set 2024-03-29T11:43:15.421 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.415+0000 7f1e1a4f6640 10 monclient: shutdown 2024-03-29T11:43:15.421 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.415+0000 7f1e1a4f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:15.421 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.415+0000 7f1e1a4f6640 1 -- 172.21.15.72:0/3569833137 >> v1:172.21.15.72:6789/0 conn(0x7f1e140b17d0 legacy=0x7f1e140b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:15.421 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.415+0000 7f1e1a4f6640 1 -- 172.21.15.72:0/3569833137 shutdown_connections 2024-03-29T11:43:15.421 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.415+0000 7f1e1a4f6640 1 -- 172.21.15.72:0/3569833137 >> 172.21.15.72:0/3569833137 conn(0x7f1e14066a00 msgr2=0x7f1e14066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:15.421 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.418+0000 7f1e1a4f6640 1 -- 172.21.15.72:0/3569833137 shutdown_connections 2024-03-29T11:43:15.422 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.418+0000 7f1e1a4f6640 1 -- 172.21.15.72:0/3569833137 wait complete. 2024-03-29T11:43:15.422 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.419+0000 7f1e1a4f6640 10 monclient: build_initial_monmap 2024-03-29T11:43:15.423 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.419+0000 7f1e1a4f6640 10 monclient: monmap: 2024-03-29T11:43:15.423 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:43:15.423 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:15.423 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:43:15.423 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:43:15.423 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:15.423 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:15.423 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:43:15.423 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:43:15.423 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:43:15.423 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:15.424 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.420+0000 7f1e1a4f6640 1 librados: starting msgr at 2024-03-29T11:43:15.424 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.420+0000 7f1e1a4f6640 1 librados: starting objecter 2024-03-29T11:43:15.424 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.420+0000 7f1e1a4f6640 1 Processor -- start 2024-03-29T11:43:15.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.421+0000 7f1e1a4f6640 1 -- start start 2024-03-29T11:43:15.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.421+0000 7f1e1a4f6640 1 librados: setting wanted keys 2024-03-29T11:43:15.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.421+0000 7f1e1a4f6640 1 librados: calling monclient init 2024-03-29T11:43:15.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.421+0000 7f1e1a4f6640 10 monclient: init 2024-03-29T11:43:15.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.422+0000 7f1e1a4f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:15.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.422+0000 7f1e1a4f6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:43:15.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.424+0000 7f1e1a4f6640 10 monclient(hunting): picked mon.noname-a con 0x7f1e140b17d0 addr v1:172.21.15.72:6789/0 2024-03-29T11:43:15.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.424+0000 7f1e13fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f1e140b17d0 0x7f1e1414bb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45387/0 (socket says 172.21.15.72:45387) 2024-03-29T11:43:15.432 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.428+0000 7f1e1a4f6640 10 monclient(hunting): picked mon.noname-c con 0x7f1e14142690 addr v1:172.21.15.72:6790/0 2024-03-29T11:43:15.432 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.429+0000 7f1e1a4f6640 10 monclient(hunting): picked mon.noname-b con 0x7f1e1414d8b0 addr v1:172.21.15.89:6789/0 2024-03-29T11:43:15.432 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.429+0000 7f1e1a4f6640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e140b64a0 con 0x7f1e140b17d0 2024-03-29T11:43:15.432 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.429+0000 7f1e1a4f6640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e140b6710 con 0x7f1e1414d8b0 2024-03-29T11:43:15.432 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.429+0000 7f1e1a4f6640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e1414dd40 con 0x7f1e14142690 2024-03-29T11:43:15.432 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.429+0000 7f1e1a4f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:15.432 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.429+0000 7f1e1a4f6640 10 monclient(hunting): authenticate will time out at 1354.717533s 2024-03-29T11:43:15.434 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.430+0000 7f1e13fff640 1 -- 172.21.15.72:0/3431952311 learned_addr learned my addr 172.21.15.72:0/3431952311 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:43:15.436 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.431+0000 7f1e10ff9640 10 client.?.objecter ms_handle_connect 0x7f1e140b17d0 2024-03-29T11:43:15.436 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.431+0000 7f1e10ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:15.436 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.431+0000 7f1e10ff9640 10 client.?.objecter ms_handle_connect 0x7f1e14142690 2024-03-29T11:43:15.436 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.431+0000 7f1e10ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:15.436 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.431+0000 7f1e10ff9640 1 -- 172.21.15.72:0/3431952311 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2228817852 0 0) 0x7f1e140b64a0 con 0x7f1e140b17d0 2024-03-29T11:43:15.436 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.431+0000 7f1e10ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:43:15.436 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.431+0000 7f1e10ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:15.436 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.431+0000 7f1e10ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:15.437 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.431+0000 7f1e10ff9640 10 monclient(hunting): my global_id is 4521 2024-03-29T11:43:15.437 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.431+0000 7f1e10ff9640 1 -- 172.21.15.72:0/3431952311 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e00004eb0 con 0x7f1e140b17d0 2024-03-29T11:43:15.437 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.432+0000 7f1e10ff9640 1 -- 172.21.15.72:0/3431952311 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3785864620 0 0) 0x7f1e1414dd40 con 0x7f1e14142690 2024-03-29T11:43:15.437 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.432+0000 7f1e10ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:43:15.437 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.432+0000 7f1e10ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:15.437 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.432+0000 7f1e10ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:15.437 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.432+0000 7f1e10ff9640 10 monclient(hunting): my global_id is 4535 2024-03-29T11:43:15.437 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.432+0000 7f1e10ff9640 1 -- 172.21.15.72:0/3431952311 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e140b64a0 con 0x7f1e14142690 2024-03-29T11:43:15.437 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.432+0000 7f1e10ff9640 10 client.?.objecter ms_handle_connect 0x7f1e1414d8b0 2024-03-29T11:43:15.437 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.432+0000 7f1e10ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:15.437 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.433+0000 7f1e10ff9640 1 -- 172.21.15.72:0/3431952311 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1429311562 0 0) 0x7f1e140b6710 con 0x7f1e1414d8b0 2024-03-29T11:43:15.437 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.433+0000 7f1e10ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:43:15.437 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.433+0000 7f1e10ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:15.437 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.433+0000 7f1e10ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:15.437 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.433+0000 7f1e10ff9640 10 monclient(hunting): my global_id is 4465 2024-03-29T11:43:15.437 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.433+0000 7f1e10ff9640 1 -- 172.21.15.72:0/3431952311 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e1414dd40 con 0x7f1e1414d8b0 2024-03-29T11:43:15.438 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.434+0000 7f1e10ff9640 1 -- 172.21.15.72:0/3431952311 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2322608649 0 0) 0x7f1e1414dd40 con 0x7f1e1414d8b0 2024-03-29T11:43:15.438 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.434+0000 7f1e10ff9640 1 -- 172.21.15.72:0/3431952311 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1e140b6710 con 0x7f1e1414d8b0 2024-03-29T11:43:15.438 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.434+0000 7f1e10ff9640 1 -- 172.21.15.72:0/3431952311 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f1e0c001e60 con 0x7f1e1414d8b0 2024-03-29T11:43:15.438 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.434+0000 7f1e10ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:15.438 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.434+0000 7f1e10ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:15.438 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.434+0000 7f1e10ff9640 10 monclient(hunting): dump: 2024-03-29T11:43:15.438 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:43:15.438 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:15.438 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:15.438 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:15.439 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:15.439 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:15.439 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:43:15.439 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:43:15.439 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:43:15.439 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:15.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.434+0000 7f1e10ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:15.441 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.435+0000 7f1e10ff9640 1 -- 172.21.15.72:0/3431952311 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 288375735 0 0) 0x7f1e00004eb0 con 0x7f1e140b17d0 2024-03-29T11:43:15.441 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.436+0000 7f1e10ff9640 1 -- 172.21.15.72:0/3431952311 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1e1414dd40 con 0x7f1e140b17d0 2024-03-29T11:43:15.441 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.436+0000 7f1e10ff9640 1 -- 172.21.15.72:0/3431952311 <== mon.1 v1:172.21.15.89:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3000245638 0 0) 0x7f1e140b6710 con 0x7f1e1414d8b0 2024-03-29T11:43:15.441 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.436+0000 7f1e10ff9640 1 -- 172.21.15.72:0/3431952311 >> v1:172.21.15.72:6790/0 conn(0x7f1e14142690 legacy=0x7f1e1412ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:15.441 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.436+0000 7f1e10ff9640 1 -- 172.21.15.72:0/3431952311 >> v1:172.21.15.72:6789/0 conn(0x7f1e140b17d0 legacy=0x7f1e1414bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:15.441 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.436+0000 7f1e10ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:43:15.441 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.436+0000 7f1e10ff9640 1 monclient: found mon.b 2024-03-29T11:43:15.441 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.436+0000 7f1e10ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:15.442 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.436+0000 7f1e10ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:43:15.442 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.436+0000 7f1e10ff9640 1 -- 172.21.15.72:0/3431952311 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e1414c280 con 0x7f1e1414d8b0 2024-03-29T11:43:15.442 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.436+0000 7f1e10ff9640 10 monclient: _finish_auth 0 2024-03-29T11:43:15.442 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.436+0000 7f1e10ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:43:15.442 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.436+0000 7f1e10ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:15.442 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.437+0000 7f1e1a4f6640 5 monclient: authenticate success, global_id 4465 2024-03-29T11:43:15.442 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.437+0000 7f1e1a4f6640 10 monclient: _renew_subs 2024-03-29T11:43:15.442 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.437+0000 7f1e1a4f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:43:15.442 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.437+0000 7f1e1a4f6640 1 -- 172.21.15.72:0/3431952311 --> v1:172.21.15.89:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e1414d450 con 0x7f1e1414d8b0 2024-03-29T11:43:15.442 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.437+0000 7f1e1a4f6640 10 client.4465.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:15.442 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.437+0000 7f1e1a4f6640 10 monclient: _renew_subs 2024-03-29T11:43:15.442 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.437+0000 7f1e1a4f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:43:15.442 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.437+0000 7f1e1a4f6640 1 -- 172.21.15.72:0/3431952311 --> v1:172.21.15.89:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e1414e440 con 0x7f1e1414d8b0 2024-03-29T11:43:15.442 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.437+0000 7f1e1a4f6640 1 librados: init done 2024-03-29T11:43:15.442 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.437+0000 7f1e10ff9640 1 -- 172.21.15.72:0/3431952311 <== mon.1 v1:172.21.15.89:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1e0c003c20 con 0x7f1e1414d8b0 2024-03-29T11:43:15.442 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.437+0000 7f1e10ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:15.443 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.437+0000 7f1e127fc640 4 set_mon_vals no callback set 2024-03-29T11:43:15.443 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.437+0000 7f1e10ff9640 1 -- 172.21.15.72:0/3431952311 <== mon.1 v1:172.21.15.89:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f1e0c003f30 con 0x7f1e1414d8b0 2024-03-29T11:43:15.443 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.437+0000 7f1e10ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:15.443 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.437+0000 7f1e10ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:15.443 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.437+0000 7f1e10ff9640 10 monclient: dump: 2024-03-29T11:43:15.443 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:43:15.443 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:15.443 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:15.443 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:15.443 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:15.443 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:15.443 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:43:15.443 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:43:15.443 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:43:15.443 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:15.444 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.439+0000 7f1e1a4f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:15.444 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.439+0000 7f1e1a4f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:15.444 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.439+0000 7f1e1a4f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:43:15.444 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.439+0000 7f1e1a4f6640 1 -- 172.21.15.72:0/3431952311 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e14075f10 con 0x7f1e1414d8b0 2024-03-29T11:43:15.447 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.442+0000 7f1e10ff9640 1 -- 172.21.15.72:0/3431952311 <== mon.1 v1:172.21.15.89:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f1e0c019f00 con 0x7f1e1414d8b0 2024-03-29T11:43:15.447 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.443+0000 7f1e10ff9640 1 -- 172.21.15.72:0/3431952311 <== mon.1 v1:172.21.15.89:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f1e0c083a90 con 0x7f1e1414d8b0 2024-03-29T11:43:15.447 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.443+0000 7f1e10ff9640 10 client.4465.objecter ms_dispatch 0x7f1e14000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:15.447 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.443+0000 7f1e10ff9640 3 client.4465.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:15.447 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.443+0000 7f1e10ff9640 3 client.4465.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:15.447 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.443+0000 7f1e10ff9640 20 client.4465.objecter dump_active .. 0 homeless 2024-03-29T11:43:15.447 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.443+0000 7f1e13fff640 10 monclient: get_auth_request con 0x7f1e0006dd80 auth_method 0 2024-03-29T11:43:15.448 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.444+0000 7f1e10ff9640 10 client.4465.objecter ms_handle_connect 0x7f1e0006dd80 2024-03-29T11:43:15.450 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.447+0000 7f1e10ff9640 1 -- 172.21.15.72:0/3431952311 <== mon.1 v1:172.21.15.89:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1e0c04f370 con 0x7f1e1414d8b0 2024-03-29T11:43:15.450 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.447+0000 7f1e10ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:15.450 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.447+0000 7f1e10ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:15.777 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.774+0000 7f1e1a4f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:15.778 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.774+0000 7f1e1a4f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:15.778 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.774+0000 7f1e1a4f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:43:15.778 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.774+0000 7f1e1a4f6640 1 -- 172.21.15.72:0/3431952311 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e1414c410 con 0x7f1e1414d8b0 2024-03-29T11:43:15.782 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.779+0000 7f1e10ff9640 1 -- 172.21.15.72:0/3431952311 <== mon.1 v1:172.21.15.89:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f1e0c0512a0 con 0x7f1e1414d8b0 2024-03-29T11:43:15.782 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.779+0000 7f1e10ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:15.782 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.779+0000 7f1e10ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:15.784 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:43:15.784 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:15.788 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.784+0000 7f1e1a4f6640 10 librados: watch_flush enter 2024-03-29T11:43:15.788 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.784+0000 7f1e1a4f6640 10 librados: watch_flush exit 2024-03-29T11:43:15.788 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.784+0000 7f1e1a4f6640 20 client.4465.objecter shutdown clearing up homeless session... 2024-03-29T11:43:15.788 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.784+0000 7f1e1a4f6640 10 client.4465.objecter successfully canceled tick 2024-03-29T11:43:15.788 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.784+0000 7f1e1a4f6640 1 -- 172.21.15.72:0/3431952311 >> v1:172.21.15.72:6816/1578866102 conn(0x7f1e0006dd80 legacy=0x7f1e00070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:15.788 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.784+0000 7f1e1a4f6640 10 monclient: shutdown 2024-03-29T11:43:15.789 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.784+0000 7f1e1a4f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:15.789 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.784+0000 7f1e1a4f6640 1 -- 172.21.15.72:0/3431952311 >> v1:172.21.15.89:6789/0 conn(0x7f1e1414d8b0 legacy=0x7f1e141e6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:15.789 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.785+0000 7f1e1a4f6640 1 -- 172.21.15.72:0/3431952311 shutdown_connections 2024-03-29T11:43:15.789 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.785+0000 7f1e1a4f6640 1 -- 172.21.15.72:0/3431952311 >> 172.21.15.72:0/3431952311 conn(0x7f1e14066a00 msgr2=0x7f1e14066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:15.789 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.785+0000 7f1e1a4f6640 1 -- 172.21.15.72:0/3431952311 shutdown_connections 2024-03-29T11:43:15.792 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.786+0000 7f1e1a4f6640 1 -- 172.21.15.72:0/3431952311 wait complete. 2024-03-29T11:43:15.792 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:15.787+0000 7f1e1a4f6640 1 librados: shutdown 2024-03-29T11:43:15.808 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:22.615 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:22.784 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.780+0000 7ff2b6d4b640 10 monclient: get_monmap_and_config 2024-03-29T11:43:22.784 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.780+0000 7ff2b6d4b640 10 monclient: build_initial_monmap 2024-03-29T11:43:22.784 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.780+0000 7ff2b6d4b640 10 monclient: monmap: 2024-03-29T11:43:22.784 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:43:22.784 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:22.784 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:43:22.781690+0000 2024-03-29T11:43:22.784 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:43:22.781690+0000 2024-03-29T11:43:22.784 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:22.784 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:22.784 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:43:22.785 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:43:22.785 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:43:22.785 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:22.785 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.781+0000 7ff2b6d4b640 1 Processor -- start 2024-03-29T11:43:22.785 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.781+0000 7ff2b6d4b640 1 -- start start 2024-03-29T11:43:22.785 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.781+0000 7ff2b6d4b640 10 monclient: init 2024-03-29T11:43:22.791 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.782+0000 7ff2b6d4b640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:22.791 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.782+0000 7ff2b6d4b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:43:22.791 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.782+0000 7ff2b6d4b640 10 monclient(hunting): picked mon.noname-b con 0x7ff2b01457f0 addr v1:172.21.15.72:6790/0 2024-03-29T11:43:22.791 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.783+0000 7ff2b6d4b640 10 monclient(hunting): picked mon.noname-a con 0x7ff2b013f650 addr v1:172.21.15.72:6789/0 2024-03-29T11:43:22.791 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.783+0000 7ff2b4ae7640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7ff2b01457f0 0x7ff2b013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:44090/0 (socket says 172.21.15.72:44090) 2024-03-29T11:43:22.791 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.783+0000 7ff2b4ae7640 1 -- 172.21.15.72:0/2176552699 learned_addr learned my addr 172.21.15.72:0/2176552699 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:43:22.792 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.783+0000 7ff2b6d4b640 10 monclient(hunting): picked mon.noname-c con 0x7ff2b00a9d70 addr v1:172.21.15.89:6789/0 2024-03-29T11:43:22.792 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.783+0000 7ff2b6d4b640 1 -- 172.21.15.72:0/2176552699 --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2b00aaef0 con 0x7ff2b013f650 2024-03-29T11:43:22.792 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.783+0000 7ff2b6d4b640 1 -- 172.21.15.72:0/2176552699 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2b0145bd0 con 0x7ff2b00a9d70 2024-03-29T11:43:22.792 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.783+0000 7ff2b6d4b640 1 -- 172.21.15.72:0/2176552699 --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2b0145eb0 con 0x7ff2b01457f0 2024-03-29T11:43:22.792 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.783+0000 7ff2b6d4b640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:22.792 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.783+0000 7ff2b6d4b640 10 monclient(hunting): authenticate will time out at 1362.071689s 2024-03-29T11:43:22.792 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.783+0000 7ff2af7fe640 1 -- 172.21.15.72:0/2176552699 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1485929359 0 0) 0x7ff2b0145bd0 con 0x7ff2b00a9d70 2024-03-29T11:43:22.792 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.783+0000 7ff2af7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:43:22.792 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.784+0000 7ff2af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:22.792 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.784+0000 7ff2af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:22.792 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.784+0000 7ff2af7fe640 10 monclient(hunting): my global_id is 4468 2024-03-29T11:43:22.792 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.784+0000 7ff2af7fe640 1 -- 172.21.15.72:0/2176552699 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff29c003520 con 0x7ff2b00a9d70 2024-03-29T11:43:22.792 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.784+0000 7ff2af7fe640 1 -- 172.21.15.72:0/2176552699 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2360614218 0 0) 0x7ff2b0145eb0 con 0x7ff2b01457f0 2024-03-29T11:43:22.792 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.784+0000 7ff2af7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:43:22.792 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.784+0000 7ff2af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:22.793 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.784+0000 7ff2af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:22.793 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.784+0000 7ff2af7fe640 10 monclient(hunting): my global_id is 4538 2024-03-29T11:43:22.793 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.784+0000 7ff2af7fe640 1 -- 172.21.15.72:0/2176552699 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2b0145bd0 con 0x7ff2b01457f0 2024-03-29T11:43:22.793 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.784+0000 7ff2af7fe640 1 -- 172.21.15.72:0/2176552699 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1396855778 0 0) 0x7ff2b00aaef0 con 0x7ff2b013f650 2024-03-29T11:43:22.793 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.784+0000 7ff2af7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:43:22.793 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.784+0000 7ff2af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:22.793 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.784+0000 7ff2af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:22.793 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.784+0000 7ff2af7fe640 10 monclient(hunting): my global_id is 4524 2024-03-29T11:43:22.793 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.784+0000 7ff2af7fe640 1 -- 172.21.15.72:0/2176552699 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2b0145eb0 con 0x7ff2b013f650 2024-03-29T11:43:22.793 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.784+0000 7ff2af7fe640 1 -- 172.21.15.72:0/2176552699 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1105300979 0 0) 0x7ff29c003520 con 0x7ff2b00a9d70 2024-03-29T11:43:22.793 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.784+0000 7ff2af7fe640 1 -- 172.21.15.72:0/2176552699 >> v1:172.21.15.72:6790/0 conn(0x7ff2b01457f0 legacy=0x7ff2b013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:22.793 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.784+0000 7ff2af7fe640 1 -- 172.21.15.72:0/2176552699 >> v1:172.21.15.72:6789/0 conn(0x7ff2b013f650 legacy=0x7ff2b00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:22.793 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.784+0000 7ff2af7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:43:22.793 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.784+0000 7ff2af7fe640 1 monclient: found mon.noname-c 2024-03-29T11:43:22.794 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.784+0000 7ff2af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:22.794 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.784+0000 7ff2af7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.89:6789/0 2024-03-29T11:43:22.794 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.784+0000 7ff2af7fe640 1 -- 172.21.15.72:0/2176552699 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2b0146340 con 0x7ff2b00a9d70 2024-03-29T11:43:22.794 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.784+0000 7ff2af7fe640 10 monclient: _finish_auth 0 2024-03-29T11:43:22.794 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.784+0000 7ff2af7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:43:22.794 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.784+0000 7ff2af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:22.794 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.785+0000 7ff2af7fe640 1 -- 172.21.15.72:0/2176552699 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7ff2a8001a10 con 0x7ff2b00a9d70 2024-03-29T11:43:22.794 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.785+0000 7ff2b6d4b640 5 monclient: authenticate success, global_id 4468 2024-03-29T11:43:22.794 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.785+0000 7ff2b6d4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:22.794 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.785+0000 7ff2af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:22.794 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.785+0000 7ff2af7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:22.794 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.785+0000 7ff2af7fe640 10 monclient: dump: 2024-03-29T11:43:22.794 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:43:22.794 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:22.794 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:22.795 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:22.795 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:22.795 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:22.795 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:43:22.795 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:43:22.795 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:43:22.795 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:22.795 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.785+0000 7ff2b6d4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:22.795 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.785+0000 7ff2af7fe640 1 -- 172.21.15.72:0/2176552699 <== mon.1 v1:172.21.15.89:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff2a8001b70 con 0x7ff2b00a9d70 2024-03-29T11:43:22.795 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.785+0000 7ff2af7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:22.795 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.785+0000 7ff2af7fe640 1 -- 172.21.15.72:0/2176552699 <== mon.1 v1:172.21.15.89:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7ff2a8001f80 con 0x7ff2b00a9d70 2024-03-29T11:43:22.795 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.785+0000 7ff2b6d4b640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:22.795 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.785+0000 7ff2b6d4b640 4 set_mon_vals no callback set 2024-03-29T11:43:22.795 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.786+0000 7ff2af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:22.795 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.786+0000 7ff2af7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:22.795 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.786+0000 7ff2af7fe640 10 monclient: dump: 2024-03-29T11:43:22.796 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:43:22.796 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:22.796 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:22.796 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:22.796 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:22.796 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:22.796 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:43:22.796 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:43:22.796 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:43:22.796 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:22.796 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.786+0000 7ff2b6d4b640 10 monclient: shutdown 2024-03-29T11:43:22.796 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.786+0000 7ff2b6d4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:22.796 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.786+0000 7ff2b6d4b640 1 -- 172.21.15.72:0/2176552699 >> v1:172.21.15.89:6789/0 conn(0x7ff2b00a9d70 legacy=0x7ff2b014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:22.796 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.786+0000 7ff2b6d4b640 1 -- 172.21.15.72:0/2176552699 shutdown_connections 2024-03-29T11:43:22.796 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.787+0000 7ff2b6d4b640 1 -- 172.21.15.72:0/2176552699 >> 172.21.15.72:0/2176552699 conn(0x7ff2b0066a00 msgr2=0x7ff2b0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:22.797 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.787+0000 7ff2b6d4b640 1 -- 172.21.15.72:0/2176552699 shutdown_connections 2024-03-29T11:43:22.797 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.787+0000 7ff2b6d4b640 1 -- 172.21.15.72:0/2176552699 wait complete. 2024-03-29T11:43:22.797 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.788+0000 7ff2b6d4b640 10 monclient: build_initial_monmap 2024-03-29T11:43:22.797 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.788+0000 7ff2b6d4b640 10 monclient: monmap: 2024-03-29T11:43:22.797 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:43:22.797 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:22.797 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:43:22.797 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:43:22.797 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:22.797 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:22.797 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:43:22.797 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:43:22.797 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:43:22.797 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:22.805 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.789+0000 7ff2b6d4b640 1 librados: starting msgr at 2024-03-29T11:43:22.805 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.789+0000 7ff2b6d4b640 1 librados: starting objecter 2024-03-29T11:43:22.805 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.789+0000 7ff2b6d4b640 1 Processor -- start 2024-03-29T11:43:22.805 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.789+0000 7ff2b6d4b640 1 -- start start 2024-03-29T11:43:22.805 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.789+0000 7ff2b6d4b640 1 librados: setting wanted keys 2024-03-29T11:43:22.805 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.789+0000 7ff2b6d4b640 1 librados: calling monclient init 2024-03-29T11:43:22.805 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.789+0000 7ff2b6d4b640 10 monclient: init 2024-03-29T11:43:22.805 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.790+0000 7ff2b6d4b640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:22.806 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.790+0000 7ff2b6d4b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:43:22.806 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.790+0000 7ff2b6d4b640 10 monclient(hunting): picked mon.noname-c con 0x7ff2b00a9d70 addr v1:172.21.15.72:6790/0 2024-03-29T11:43:22.806 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.790+0000 7ff2b6d4b640 10 monclient(hunting): picked mon.noname-b con 0x7ff2b013f650 addr v1:172.21.15.89:6789/0 2024-03-29T11:43:22.806 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.791+0000 7ff2b6d4b640 10 monclient(hunting): picked mon.noname-a con 0x7ff2b01457f0 addr v1:172.21.15.72:6789/0 2024-03-29T11:43:22.806 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.791+0000 7ff2b6d4b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2b01cbd90 con 0x7ff2b01457f0 2024-03-29T11:43:22.806 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.791+0000 7ff2b52e8640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7ff2b01457f0 0x7ff2b013fda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:59796/0 (socket says 172.21.15.72:59796) 2024-03-29T11:43:22.806 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.791+0000 7ff2b52e8640 1 -- 172.21.15.72:0/3964922595 learned_addr learned my addr 172.21.15.72:0/3964922595 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:43:22.806 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.791+0000 7ff2b6d4b640 1 -- 172.21.15.72:0/3964922595 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2b01ccef0 con 0x7ff2b013f650 2024-03-29T11:43:22.806 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.791+0000 7ff2b6d4b640 1 -- 172.21.15.72:0/3964922595 --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2b0142a60 con 0x7ff2b00a9d70 2024-03-29T11:43:22.806 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.791+0000 7ff2b6d4b640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:22.806 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.791+0000 7ff2b6d4b640 10 monclient(hunting): authenticate will time out at 1362.079448s 2024-03-29T11:43:22.806 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.791+0000 7ff2ad7fa640 10 client.?.objecter ms_handle_connect 0x7ff2b013f650 2024-03-29T11:43:22.806 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.791+0000 7ff2ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:22.807 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.791+0000 7ff2ad7fa640 10 client.?.objecter ms_handle_connect 0x7ff2b01457f0 2024-03-29T11:43:22.807 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.791+0000 7ff2ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:22.807 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.791+0000 7ff2ad7fa640 1 -- 172.21.15.72:0/3964922595 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2164059124 0 0) 0x7ff2b01cbd90 con 0x7ff2b01457f0 2024-03-29T11:43:22.807 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.791+0000 7ff2ad7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:43:22.807 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.791+0000 7ff2ad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:22.807 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.791+0000 7ff2ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:22.807 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.791+0000 7ff2ad7fa640 10 monclient(hunting): my global_id is 4527 2024-03-29T11:43:22.807 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.791+0000 7ff2ad7fa640 1 -- 172.21.15.72:0/3964922595 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff290004eb0 con 0x7ff2b01457f0 2024-03-29T11:43:22.807 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.791+0000 7ff2ad7fa640 10 client.?.objecter ms_handle_connect 0x7ff2b00a9d70 2024-03-29T11:43:22.807 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.791+0000 7ff2ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:22.807 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.791+0000 7ff2ad7fa640 1 -- 172.21.15.72:0/3964922595 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2869593471 0 0) 0x7ff2b01ccef0 con 0x7ff2b013f650 2024-03-29T11:43:22.807 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.791+0000 7ff2ad7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:43:22.807 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.791+0000 7ff2ad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:22.807 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.791+0000 7ff2ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:22.808 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.791+0000 7ff2ad7fa640 10 monclient(hunting): my global_id is 4471 2024-03-29T11:43:22.808 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.791+0000 7ff2ad7fa640 1 -- 172.21.15.72:0/3964922595 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2b01cbd90 con 0x7ff2b013f650 2024-03-29T11:43:22.808 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.792+0000 7ff2ad7fa640 1 -- 172.21.15.72:0/3964922595 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2176201634 0 0) 0x7ff2b0142a60 con 0x7ff2b00a9d70 2024-03-29T11:43:22.808 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.792+0000 7ff2ad7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:43:22.808 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.792+0000 7ff2ad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:22.808 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.792+0000 7ff2ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:22.808 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.792+0000 7ff2ad7fa640 10 monclient(hunting): my global_id is 4541 2024-03-29T11:43:22.808 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.792+0000 7ff2ad7fa640 1 -- 172.21.15.72:0/3964922595 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2b01ccef0 con 0x7ff2b00a9d70 2024-03-29T11:43:22.808 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.792+0000 7ff2ad7fa640 1 -- 172.21.15.72:0/3964922595 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1603669847 0 0) 0x7ff2b01cbd90 con 0x7ff2b013f650 2024-03-29T11:43:22.808 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.792+0000 7ff2ad7fa640 1 -- 172.21.15.72:0/3964922595 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2b0142a60 con 0x7ff2b013f650 2024-03-29T11:43:22.808 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.793+0000 7ff2ad7fa640 1 -- 172.21.15.72:0/3964922595 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2711427269 0 0) 0x7ff290004eb0 con 0x7ff2b01457f0 2024-03-29T11:43:22.808 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.793+0000 7ff2ad7fa640 1 -- 172.21.15.72:0/3964922595 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2b01cbd90 con 0x7ff2b01457f0 2024-03-29T11:43:22.808 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.793+0000 7ff2ad7fa640 1 -- 172.21.15.72:0/3964922595 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7ff2a0001820 con 0x7ff2b013f650 2024-03-29T11:43:22.808 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.793+0000 7ff2ad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:22.808 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.793+0000 7ff2ad7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:22.809 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.793+0000 7ff2ad7fa640 10 monclient(hunting): dump: 2024-03-29T11:43:22.809 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:43:22.809 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:22.809 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:22.809 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:22.809 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:22.809 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:22.809 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:43:22.809 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:43:22.809 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:43:22.809 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:22.809 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.793+0000 7ff2ad7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:22.809 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.793+0000 7ff2ad7fa640 1 -- 172.21.15.72:0/3964922595 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7ff2a8001ec0 con 0x7ff2b01457f0 2024-03-29T11:43:22.809 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.793+0000 7ff2ad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:22.810 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.793+0000 7ff2ad7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:22.810 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.793+0000 7ff2ad7fa640 10 monclient(hunting): dump: 2024-03-29T11:43:22.810 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:43:22.810 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:22.810 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:22.810 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:22.810 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:22.810 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:22.810 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:43:22.810 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:43:22.810 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:43:22.810 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:22.810 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.793+0000 7ff2ad7fa640 1 -- 172.21.15.72:0/3964922595 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3748893345 0 0) 0x7ff2b01ccef0 con 0x7ff2b00a9d70 2024-03-29T11:43:22.810 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.793+0000 7ff2ad7fa640 1 -- 172.21.15.72:0/3964922595 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff290004eb0 con 0x7ff2b00a9d70 2024-03-29T11:43:22.811 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.793+0000 7ff2ad7fa640 1 -- 172.21.15.72:0/3964922595 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7ff298001770 con 0x7ff2b00a9d70 2024-03-29T11:43:22.811 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.793+0000 7ff2ad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:22.811 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.793+0000 7ff2ad7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:22.811 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.793+0000 7ff2ad7fa640 10 monclient(hunting): dump: 2024-03-29T11:43:22.811 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:43:22.811 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:22.811 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:22.811 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:22.811 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:22.811 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:22.811 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:43:22.811 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:43:22.811 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:43:22.812 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:22.812 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.793+0000 7ff2ad7fa640 1 -- 172.21.15.72:0/3964922595 <== mon.1 v1:172.21.15.89:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 268037878 0 0) 0x7ff2b0142a60 con 0x7ff2b013f650 2024-03-29T11:43:22.812 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.794+0000 7ff2ad7fa640 1 -- 172.21.15.72:0/3964922595 >> v1:172.21.15.72:6790/0 conn(0x7ff2b00a9d70 legacy=0x7ff2b01cb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:22.812 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.794+0000 7ff2ad7fa640 1 -- 172.21.15.72:0/3964922595 >> v1:172.21.15.72:6789/0 conn(0x7ff2b01457f0 legacy=0x7ff2b013fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:22.812 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.794+0000 7ff2ad7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:43:22.812 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.794+0000 7ff2ad7fa640 1 monclient: found mon.b 2024-03-29T11:43:22.812 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.794+0000 7ff2ad7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:22.812 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.794+0000 7ff2ad7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:43:22.812 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.794+0000 7ff2ad7fa640 1 -- 172.21.15.72:0/3964922595 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2b01cba50 con 0x7ff2b013f650 2024-03-29T11:43:22.812 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.794+0000 7ff2ad7fa640 10 monclient: _finish_auth 0 2024-03-29T11:43:22.812 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.794+0000 7ff2ad7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:43:22.813 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.794+0000 7ff2ad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:22.813 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.794+0000 7ff2b6d4b640 5 monclient: authenticate success, global_id 4471 2024-03-29T11:43:22.813 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.794+0000 7ff2b6d4b640 10 monclient: _renew_subs 2024-03-29T11:43:22.813 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.794+0000 7ff2b6d4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:43:22.813 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.794+0000 7ff2b6d4b640 1 -- 172.21.15.72:0/3964922595 --> v1:172.21.15.89:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2b01cef70 con 0x7ff2b013f650 2024-03-29T11:43:22.813 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.794+0000 7ff2b6d4b640 10 client.4471.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:22.813 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.794+0000 7ff2b6d4b640 10 monclient: _renew_subs 2024-03-29T11:43:22.813 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.794+0000 7ff2b6d4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:43:22.813 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.794+0000 7ff2b6d4b640 1 -- 172.21.15.72:0/3964922595 --> v1:172.21.15.89:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2b0142760 con 0x7ff2b013f650 2024-03-29T11:43:22.813 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.794+0000 7ff2b6d4b640 1 librados: init done 2024-03-29T11:43:22.813 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.795+0000 7ff2ad7fa640 1 -- 172.21.15.72:0/3964922595 <== mon.1 v1:172.21.15.89:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff2a0001d30 con 0x7ff2b013f650 2024-03-29T11:43:22.813 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.795+0000 7ff2ad7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:22.813 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.795+0000 7ff2ae7fc640 4 set_mon_vals no callback set 2024-03-29T11:43:22.813 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.796+0000 7ff2b6d4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:22.813 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.796+0000 7ff2ad7fa640 1 -- 172.21.15.72:0/3964922595 <== mon.1 v1:172.21.15.89:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7ff2a00037b0 con 0x7ff2b013f650 2024-03-29T11:43:22.814 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.796+0000 7ff2ad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:22.814 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.796+0000 7ff2ad7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:22.814 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.796+0000 7ff2ad7fa640 10 monclient: dump: 2024-03-29T11:43:22.814 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:43:22.814 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:22.814 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:22.814 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:22.814 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:22.814 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:22.814 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:43:22.814 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:43:22.814 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:43:22.814 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:22.814 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.798+0000 7ff2b6d4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:22.814 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.798+0000 7ff2b6d4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:43:22.815 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.798+0000 7ff2b6d4b640 1 -- 172.21.15.72:0/3964922595 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff294005080 con 0x7ff2b013f650 2024-03-29T11:43:22.815 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.798+0000 7ff2ad7fa640 1 -- 172.21.15.72:0/3964922595 <== mon.1 v1:172.21.15.89:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7ff2a0019780 con 0x7ff2b013f650 2024-03-29T11:43:22.815 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.798+0000 7ff2ad7fa640 1 -- 172.21.15.72:0/3964922595 <== mon.1 v1:172.21.15.89:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7ff2a00837b0 con 0x7ff2b013f650 2024-03-29T11:43:22.815 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.798+0000 7ff2ad7fa640 10 client.4471.objecter ms_dispatch 0x7ff2b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:22.815 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.798+0000 7ff2ad7fa640 3 client.4471.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:22.815 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.798+0000 7ff2ad7fa640 3 client.4471.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:22.815 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.799+0000 7ff2ad7fa640 20 client.4471.objecter dump_active .. 0 homeless 2024-03-29T11:43:22.815 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.799+0000 7ff2b4ae7640 10 monclient: get_auth_request con 0x7ff29006e0e0 auth_method 0 2024-03-29T11:43:22.815 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.799+0000 7ff2ad7fa640 10 client.4471.objecter ms_handle_connect 0x7ff29006e0e0 2024-03-29T11:43:22.816 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.804+0000 7ff2ad7fa640 1 -- 172.21.15.72:0/3964922595 <== mon.1 v1:172.21.15.89:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff2a004f090 con 0x7ff2b013f650 2024-03-29T11:43:22.816 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.804+0000 7ff2ad7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:22.816 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:22.804+0000 7ff2ad7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:23.214 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:23.210+0000 7ff2b6d4b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:23.214 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:23.210+0000 7ff2b6d4b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:23.214 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:23.210+0000 7ff2b6d4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:43:23.214 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:23.210+0000 7ff2b6d4b640 1 -- 172.21.15.72:0/3964922595 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff294005330 con 0x7ff2b013f650 2024-03-29T11:43:23.215 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:23.212+0000 7ff2ad7fa640 1 -- 172.21.15.72:0/3964922595 <== mon.1 v1:172.21.15.89:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7ff2a0050fc0 con 0x7ff2b013f650 2024-03-29T11:43:23.216 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:23.212+0000 7ff2ad7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:23.216 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:23.212+0000 7ff2ad7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:23.216 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:43:23.216 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:23.221 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:23.216+0000 7ff2b6d4b640 10 librados: watch_flush enter 2024-03-29T11:43:23.221 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:23.216+0000 7ff2b6d4b640 10 librados: watch_flush exit 2024-03-29T11:43:23.221 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:23.216+0000 7ff2b6d4b640 20 client.4471.objecter shutdown clearing up homeless session... 2024-03-29T11:43:23.221 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:23.216+0000 7ff2b6d4b640 10 client.4471.objecter successfully canceled tick 2024-03-29T11:43:23.221 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:23.216+0000 7ff2b6d4b640 1 -- 172.21.15.72:0/3964922595 >> v1:172.21.15.72:6816/1578866102 conn(0x7ff29006e0e0 legacy=0x7ff2900705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:23.221 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:23.216+0000 7ff2b6d4b640 10 monclient: shutdown 2024-03-29T11:43:23.221 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:23.216+0000 7ff2b6d4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:23.221 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:23.216+0000 7ff2b6d4b640 1 -- 172.21.15.72:0/3964922595 >> v1:172.21.15.89:6789/0 conn(0x7ff2b013f650 legacy=0x7ff2b01ce860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:23.221 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:23.217+0000 7ff2b6d4b640 1 -- 172.21.15.72:0/3964922595 shutdown_connections 2024-03-29T11:43:23.221 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:23.217+0000 7ff2b6d4b640 1 -- 172.21.15.72:0/3964922595 >> 172.21.15.72:0/3964922595 conn(0x7ff2b0066a00 msgr2=0x7ff2b00b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:23.221 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:23.217+0000 7ff2b6d4b640 1 -- 172.21.15.72:0/3964922595 shutdown_connections 2024-03-29T11:43:23.221 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:23.217+0000 7ff2b6d4b640 1 -- 172.21.15.72:0/3964922595 wait complete. 2024-03-29T11:43:23.221 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:23.217+0000 7ff2b6d4b640 1 librados: shutdown 2024-03-29T11:43:23.236 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:30.044 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:30.202 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.198+0000 7fc365760640 10 monclient: get_monmap_and_config 2024-03-29T11:43:30.203 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.198+0000 7fc365760640 10 monclient: build_initial_monmap 2024-03-29T11:43:30.203 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.198+0000 7fc365760640 10 monclient: monmap: 2024-03-29T11:43:30.203 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:43:30.203 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:30.203 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:43:30.199626+0000 2024-03-29T11:43:30.203 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:43:30.199626+0000 2024-03-29T11:43:30.203 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:30.203 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:30.203 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:43:30.203 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:43:30.203 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:43:30.203 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:30.204 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.199+0000 7fc365760640 1 Processor -- start 2024-03-29T11:43:30.204 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.200+0000 7fc365760640 1 -- start start 2024-03-29T11:43:30.204 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.200+0000 7fc365760640 10 monclient: init 2024-03-29T11:43:30.204 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.200+0000 7fc365760640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:30.204 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.200+0000 7fc365760640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:43:30.204 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.200+0000 7fc365760640 10 monclient(hunting): picked mon.noname-b con 0x7fc360070450 addr v1:172.21.15.72:6790/0 2024-03-29T11:43:30.204 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.200+0000 7fc365760640 10 monclient(hunting): picked mon.noname-a con 0x7fc3600b03b0 addr v1:172.21.15.72:6789/0 2024-03-29T11:43:30.209 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.203+0000 7fc365760640 10 monclient(hunting): picked mon.noname-c con 0x7fc36013d370 addr v1:172.21.15.89:6789/0 2024-03-29T11:43:30.209 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.203+0000 7fc365760640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc36013fdc0 con 0x7fc3600b03b0 2024-03-29T11:43:30.209 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.203+0000 7fc365760640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc360071e40 con 0x7fc36013d370 2024-03-29T11:43:30.209 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.203+0000 7fc365760640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc360072120 con 0x7fc360070450 2024-03-29T11:43:30.210 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.203+0000 7fc365760640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:30.210 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.203+0000 7fc365760640 10 monclient(hunting): authenticate will time out at 1369.491644s 2024-03-29T11:43:30.210 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.203+0000 7fc364f5f640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc36013d370 0x7fc36014b520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:39227/0 (socket says 172.21.15.72:39227) 2024-03-29T11:43:30.210 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.203+0000 7fc364f5f640 1 -- 172.21.15.72:0/435026993 learned_addr learned my addr 172.21.15.72:0/435026993 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:43:30.211 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.207+0000 7fc35effd640 1 -- 172.21.15.72:0/435026993 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2159654721 0 0) 0x7fc360071e40 con 0x7fc36013d370 2024-03-29T11:43:30.211 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.207+0000 7fc35effd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:30.211 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.207+0000 7fc35effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:30.211 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.207+0000 7fc35effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:30.211 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.207+0000 7fc35effd640 10 monclient(hunting): my global_id is 4474 2024-03-29T11:43:30.211 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.207+0000 7fc35effd640 1 -- 172.21.15.72:0/435026993 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc348003520 con 0x7fc36013d370 2024-03-29T11:43:30.212 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.208+0000 7fc35effd640 1 -- 172.21.15.72:0/435026993 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 218159193 0 0) 0x7fc348003520 con 0x7fc36013d370 2024-03-29T11:43:30.212 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.208+0000 7fc35effd640 1 -- 172.21.15.72:0/435026993 >> v1:172.21.15.72:6790/0 conn(0x7fc360070450 legacy=0x7fc360140250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:43:30.212 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.208+0000 7fc35effd640 1 -- 172.21.15.72:0/435026993 >> v1:172.21.15.72:6789/0 conn(0x7fc3600b03b0 legacy=0x7fc36013cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:43:30.212 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.208+0000 7fc35effd640 10 monclient: _finish_hunting 0 2024-03-29T11:43:30.212 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.208+0000 7fc35effd640 1 monclient: found mon.noname-c 2024-03-29T11:43:30.215 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.208+0000 7fc35effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:30.215 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.208+0000 7fc35effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.89:6789/0 2024-03-29T11:43:30.215 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.208+0000 7fc35effd640 1 -- 172.21.15.72:0/435026993 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc360140960 con 0x7fc36013d370 2024-03-29T11:43:30.216 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.208+0000 7fc35effd640 10 monclient: _finish_auth 0 2024-03-29T11:43:30.216 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.208+0000 7fc35effd640 10 monclient: _check_auth_tickets 2024-03-29T11:43:30.216 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.208+0000 7fc35effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:30.216 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.209+0000 7fc365760640 5 monclient: authenticate success, global_id 4474 2024-03-29T11:43:30.216 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.209+0000 7fc365760640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:30.216 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.209+0000 7fc35effd640 1 -- 172.21.15.72:0/435026993 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fc358001a10 con 0x7fc36013d370 2024-03-29T11:43:30.216 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.209+0000 7fc35effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:30.216 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.209+0000 7fc35effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:30.216 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.209+0000 7fc35effd640 10 monclient: dump: 2024-03-29T11:43:30.216 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:43:30.216 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:30.216 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:30.216 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:30.216 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:30.217 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:30.217 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:43:30.217 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:43:30.217 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:43:30.217 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:30.217 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.209+0000 7fc365760640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:30.217 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.209+0000 7fc35effd640 1 -- 172.21.15.72:0/435026993 <== mon.1 v1:172.21.15.89:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc358001ca0 con 0x7fc36013d370 2024-03-29T11:43:30.217 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.209+0000 7fc35effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:30.217 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.209+0000 7fc365760640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:30.217 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.209+0000 7fc365760640 4 set_mon_vals no callback set 2024-03-29T11:43:30.217 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.210+0000 7fc35effd640 1 -- 172.21.15.72:0/435026993 <== mon.1 v1:172.21.15.89:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fc358002070 con 0x7fc36013d370 2024-03-29T11:43:30.217 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.210+0000 7fc35effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:30.217 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.210+0000 7fc35effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:30.217 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.210+0000 7fc35effd640 10 monclient: dump: 2024-03-29T11:43:30.217 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:43:30.217 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:30.218 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:30.218 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:30.218 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:30.218 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:30.218 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:43:30.218 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:43:30.218 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:43:30.218 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:30.218 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.210+0000 7fc365760640 10 monclient: shutdown 2024-03-29T11:43:30.218 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.210+0000 7fc365760640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:30.218 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.210+0000 7fc365760640 1 -- 172.21.15.72:0/435026993 >> v1:172.21.15.89:6789/0 conn(0x7fc36013d370 legacy=0x7fc36014b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:30.218 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.210+0000 7fc365760640 1 -- 172.21.15.72:0/435026993 shutdown_connections 2024-03-29T11:43:30.218 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.210+0000 7fc365760640 1 -- 172.21.15.72:0/435026993 >> 172.21.15.72:0/435026993 conn(0x7fc3600671c0 msgr2=0x7fc3600b59b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:30.218 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.211+0000 7fc365760640 1 -- 172.21.15.72:0/435026993 shutdown_connections 2024-03-29T11:43:30.218 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.211+0000 7fc365760640 1 -- 172.21.15.72:0/435026993 wait complete. 2024-03-29T11:43:30.219 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.212+0000 7fc365760640 10 monclient: build_initial_monmap 2024-03-29T11:43:30.219 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.212+0000 7fc365760640 10 monclient: monmap: 2024-03-29T11:43:30.219 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:43:30.220 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:30.220 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:43:30.220 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:43:30.220 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:30.220 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:30.220 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:43:30.220 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:43:30.220 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:43:30.220 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:30.222 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.213+0000 7fc365760640 1 librados: starting msgr at 2024-03-29T11:43:30.222 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.213+0000 7fc365760640 1 librados: starting objecter 2024-03-29T11:43:30.222 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.213+0000 7fc365760640 1 Processor -- start 2024-03-29T11:43:30.222 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.213+0000 7fc365760640 1 -- start start 2024-03-29T11:43:30.222 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.213+0000 7fc365760640 1 librados: setting wanted keys 2024-03-29T11:43:30.222 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.213+0000 7fc365760640 1 librados: calling monclient init 2024-03-29T11:43:30.222 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.213+0000 7fc365760640 10 monclient: init 2024-03-29T11:43:30.222 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.214+0000 7fc365760640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:30.222 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.214+0000 7fc365760640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:43:30.222 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.214+0000 7fc365760640 10 monclient(hunting): picked mon.noname-c con 0x7fc360070450 addr v1:172.21.15.72:6790/0 2024-03-29T11:43:30.222 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.218+0000 7fc365760640 10 monclient(hunting): picked mon.noname-a con 0x7fc3600b03b0 addr v1:172.21.15.72:6789/0 2024-03-29T11:43:30.224 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.220+0000 7fc365760640 10 monclient(hunting): picked mon.noname-b con 0x7fc36013d370 addr v1:172.21.15.89:6789/0 2024-03-29T11:43:30.230 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.220+0000 7fc365760640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc360135410 con 0x7fc3600b03b0 2024-03-29T11:43:30.230 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.220+0000 7fc365760640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc360137dd0 con 0x7fc36013d370 2024-03-29T11:43:30.230 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.220+0000 7fc365760640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3601df8d0 con 0x7fc360070450 2024-03-29T11:43:30.230 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.220+0000 7fc365760640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:30.230 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.220+0000 7fc365760640 10 monclient(hunting): authenticate will time out at 1369.508916s 2024-03-29T11:43:30.230 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.222+0000 7fc35f7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fc3600b03b0 0x7fc3601deab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37767/0 (socket says 172.21.15.72:37767) 2024-03-29T11:43:30.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.222+0000 7fc35f7fe640 1 -- 172.21.15.72:0/4171455856 learned_addr learned my addr 172.21.15.72:0/4171455856 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:43:30.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.223+0000 7fc35cff9640 10 client.?.objecter ms_handle_connect 0x7fc3600b03b0 2024-03-29T11:43:30.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.223+0000 7fc35cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:30.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.223+0000 7fc35cff9640 1 -- 172.21.15.72:0/4171455856 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2459589100 0 0) 0x7fc360135410 con 0x7fc3600b03b0 2024-03-29T11:43:30.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.223+0000 7fc35cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:43:30.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.223+0000 7fc35cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:30.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.223+0000 7fc35cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:30.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.223+0000 7fc35cff9640 10 monclient(hunting): my global_id is 4530 2024-03-29T11:43:30.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.223+0000 7fc35cff9640 1 -- 172.21.15.72:0/4171455856 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc34c004eb0 con 0x7fc3600b03b0 2024-03-29T11:43:30.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.224+0000 7fc35cff9640 1 -- 172.21.15.72:0/4171455856 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3555139542 0 0) 0x7fc34c004eb0 con 0x7fc3600b03b0 2024-03-29T11:43:30.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.224+0000 7fc35cff9640 1 -- 172.21.15.72:0/4171455856 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc360135410 con 0x7fc3600b03b0 2024-03-29T11:43:30.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.224+0000 7fc35cff9640 1 -- 172.21.15.72:0/4171455856 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fc350004050 con 0x7fc3600b03b0 2024-03-29T11:43:30.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.224+0000 7fc35cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:30.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.224+0000 7fc35cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:43:30.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.224+0000 7fc35cff9640 10 monclient(hunting): dump: 2024-03-29T11:43:30.231 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:43:30.232 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:30.232 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:30.232 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:30.232 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:30.232 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:30.232 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:43:30.232 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:43:30.232 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:43:30.232 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:30.232 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.224+0000 7fc35cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:30.232 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.226+0000 7fc35cff9640 10 client.?.objecter ms_handle_connect 0x7fc36013d370 2024-03-29T11:43:30.232 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.226+0000 7fc35cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:30.232 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.226+0000 7fc35cff9640 10 client.?.objecter ms_handle_connect 0x7fc360070450 2024-03-29T11:43:30.232 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.226+0000 7fc35cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:30.232 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.226+0000 7fc35cff9640 1 -- 172.21.15.72:0/4171455856 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3260705835 0 0) 0x7fc360137dd0 con 0x7fc36013d370 2024-03-29T11:43:30.234 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.226+0000 7fc35cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:43:30.234 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.226+0000 7fc35cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:30.234 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.226+0000 7fc35cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:30.235 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.226+0000 7fc35cff9640 10 monclient(hunting): my global_id is 4477 2024-03-29T11:43:30.235 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.227+0000 7fc35cff9640 1 -- 172.21.15.72:0/4171455856 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc34c004eb0 con 0x7fc36013d370 2024-03-29T11:43:30.235 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.227+0000 7fc35cff9640 1 -- 172.21.15.72:0/4171455856 <== mon.0 v1:172.21.15.72:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2180142956 0 0) 0x7fc360135410 con 0x7fc3600b03b0 2024-03-29T11:43:30.235 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.227+0000 7fc35cff9640 1 -- 172.21.15.72:0/4171455856 >> v1:172.21.15.72:6790/0 conn(0x7fc360070450 legacy=0x7fc360136ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:30.235 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.227+0000 7fc35cff9640 1 -- 172.21.15.72:0/4171455856 >> v1:172.21.15.89:6789/0 conn(0x7fc36013d370 legacy=0x7fc3601df1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:30.235 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.227+0000 7fc35cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:43:30.235 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.227+0000 7fc35cff9640 1 monclient: found mon.a 2024-03-29T11:43:30.235 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.227+0000 7fc35cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:30.235 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.227+0000 7fc35cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:43:30.235 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.227+0000 7fc35cff9640 1 -- 172.21.15.72:0/4171455856 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc360138040 con 0x7fc3600b03b0 2024-03-29T11:43:30.235 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.228+0000 7fc35cff9640 10 monclient: _finish_auth 0 2024-03-29T11:43:30.235 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.228+0000 7fc35cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:43:30.235 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.228+0000 7fc35cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:30.235 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.228+0000 7fc35cff9640 1 -- 172.21.15.72:0/4171455856 <== mon.0 v1:172.21.15.72:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3500044a0 con 0x7fc3600b03b0 2024-03-29T11:43:30.235 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.228+0000 7fc35cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:30.236 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.228+0000 7fc35e7fc640 4 set_mon_vals no callback set 2024-03-29T11:43:30.236 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.232+0000 7fc365760640 5 monclient: authenticate success, global_id 4530 2024-03-29T11:43:30.236 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.232+0000 7fc365760640 10 monclient: _renew_subs 2024-03-29T11:43:30.236 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.233+0000 7fc365760640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:43:30.236 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.233+0000 7fc35cff9640 1 -- 172.21.15.72:0/4171455856 <== mon.0 v1:172.21.15.72:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fc3500042b0 con 0x7fc3600b03b0 2024-03-29T11:43:30.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.233+0000 7fc365760640 1 -- 172.21.15.72:0/4171455856 --> v1:172.21.15.72:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc360138320 con 0x7fc3600b03b0 2024-03-29T11:43:30.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.233+0000 7fc365760640 10 client.4530.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:30.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.233+0000 7fc365760640 10 monclient: _renew_subs 2024-03-29T11:43:30.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.233+0000 7fc365760640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:43:30.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.233+0000 7fc365760640 1 -- 172.21.15.72:0/4171455856 --> v1:172.21.15.72:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc36006d150 con 0x7fc3600b03b0 2024-03-29T11:43:30.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.233+0000 7fc365760640 1 librados: init done 2024-03-29T11:43:30.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.233+0000 7fc35cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:30.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.233+0000 7fc35cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:30.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.233+0000 7fc35cff9640 10 monclient: dump: 2024-03-29T11:43:30.238 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:43:30.238 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:30.238 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:30.239 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:30.239 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:30.239 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:30.239 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:43:30.239 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:43:30.239 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:43:30.239 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:30.239 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.234+0000 7fc365760640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:30.240 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.234+0000 7fc365760640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:30.240 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.234+0000 7fc365760640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:43:30.240 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.235+0000 7fc365760640 1 -- 172.21.15.72:0/4171455856 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc36004f730 con 0x7fc3600b03b0 2024-03-29T11:43:30.240 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.236+0000 7fc35cff9640 1 -- 172.21.15.72:0/4171455856 <== mon.0 v1:172.21.15.72:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7fc350001320 con 0x7fc3600b03b0 2024-03-29T11:43:30.241 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.237+0000 7fc35cff9640 1 -- 172.21.15.72:0/4171455856 <== mon.0 v1:172.21.15.72:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7fc350084aa0 con 0x7fc3600b03b0 2024-03-29T11:43:30.241 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.237+0000 7fc35ffff640 10 monclient: get_auth_request con 0x7fc34c06da80 auth_method 0 2024-03-29T11:43:30.241 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.237+0000 7fc35cff9640 10 client.4530.objecter ms_dispatch 0x7fc360000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:30.241 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.237+0000 7fc35cff9640 3 client.4530.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:30.241 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.237+0000 7fc35cff9640 3 client.4530.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:30.242 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.238+0000 7fc35cff9640 20 client.4530.objecter dump_active .. 0 homeless 2024-03-29T11:43:30.244 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.238+0000 7fc35cff9640 10 client.4530.objecter ms_handle_connect 0x7fc34c06da80 2024-03-29T11:43:30.254 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.248+0000 7fc35cff9640 1 -- 172.21.15.72:0/4171455856 <== mon.0 v1:172.21.15.72:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc3500501b0 con 0x7fc3600b03b0 2024-03-29T11:43:30.254 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.248+0000 7fc35cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:30.254 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.248+0000 7fc35cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:30.594 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.588+0000 7fc365760640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:30.594 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.588+0000 7fc365760640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:30.594 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.588+0000 7fc365760640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:43:30.594 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.588+0000 7fc365760640 1 -- 172.21.15.72:0/4171455856 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc360074d80 con 0x7fc3600b03b0 2024-03-29T11:43:30.599 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.594+0000 7fc35cff9640 1 -- 172.21.15.72:0/4171455856 <== mon.0 v1:172.21.15.72:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7fc3500522b0 con 0x7fc3600b03b0 2024-03-29T11:43:30.599 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.594+0000 7fc35cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:30.599 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.594+0000 7fc35cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:30.600 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:43:30.600 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:30.603 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.598+0000 7fc365760640 10 librados: watch_flush enter 2024-03-29T11:43:30.603 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.598+0000 7fc365760640 10 librados: watch_flush exit 2024-03-29T11:43:30.603 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.598+0000 7fc365760640 20 client.4530.objecter shutdown clearing up homeless session... 2024-03-29T11:43:30.603 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.598+0000 7fc365760640 10 client.4530.objecter successfully canceled tick 2024-03-29T11:43:30.606 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.598+0000 7fc365760640 1 -- 172.21.15.72:0/4171455856 >> v1:172.21.15.72:6816/1578866102 conn(0x7fc34c06da80 legacy=0x7fc34c06ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:30.606 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.598+0000 7fc365760640 10 monclient: shutdown 2024-03-29T11:43:30.606 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.598+0000 7fc365760640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:30.608 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.598+0000 7fc365760640 1 -- 172.21.15.72:0/4171455856 >> v1:172.21.15.72:6789/0 conn(0x7fc3600b03b0 legacy=0x7fc3601deab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:30.608 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.598+0000 7fc365760640 1 -- 172.21.15.72:0/4171455856 shutdown_connections 2024-03-29T11:43:30.608 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.598+0000 7fc365760640 1 -- 172.21.15.72:0/4171455856 >> 172.21.15.72:0/4171455856 conn(0x7fc3600671c0 msgr2=0x7fc36013a860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:30.608 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.599+0000 7fc365760640 1 -- 172.21.15.72:0/4171455856 shutdown_connections 2024-03-29T11:43:30.608 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.599+0000 7fc365760640 1 -- 172.21.15.72:0/4171455856 wait complete. 2024-03-29T11:43:30.608 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:30.599+0000 7fc365760640 1 librados: shutdown 2024-03-29T11:43:30.627 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:37.434 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:37.592 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.587+0000 7f1d646d9640 10 monclient: get_monmap_and_config 2024-03-29T11:43:37.592 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.587+0000 7f1d646d9640 10 monclient: build_initial_monmap 2024-03-29T11:43:37.592 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.587+0000 7f1d646d9640 10 monclient: monmap: 2024-03-29T11:43:37.592 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:43:37.592 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:37.592 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:43:37.588386+0000 2024-03-29T11:43:37.592 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:43:37.588386+0000 2024-03-29T11:43:37.592 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:37.592 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:37.592 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:43:37.593 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:43:37.593 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:43:37.593 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:37.593 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.588+0000 7f1d646d9640 1 Processor -- start 2024-03-29T11:43:37.593 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.588+0000 7f1d646d9640 1 -- start start 2024-03-29T11:43:37.593 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.588+0000 7f1d646d9640 10 monclient: init 2024-03-29T11:43:37.595 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.589+0000 7f1d646d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:37.595 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.589+0000 7f1d646d9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:43:37.595 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.589+0000 7f1d646d9640 10 monclient(hunting): picked mon.noname-c con 0x7f1d5c0ae250 addr v1:172.21.15.89:6789/0 2024-03-29T11:43:37.599 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.592+0000 7f1d646d9640 10 monclient(hunting): picked mon.noname-b con 0x7f1d5c13f970 addr v1:172.21.15.72:6790/0 2024-03-29T11:43:37.599 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.592+0000 7f1d646d9640 10 monclient(hunting): picked mon.noname-a con 0x7f1d5c142800 addr v1:172.21.15.72:6789/0 2024-03-29T11:43:37.599 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.592+0000 7f1d646d9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d5c0afe00 con 0x7f1d5c142800 2024-03-29T11:43:37.599 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.592+0000 7f1d646d9640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d5c142c10 con 0x7f1d5c0ae250 2024-03-29T11:43:37.599 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.592+0000 7f1d646d9640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d5c0afb20 con 0x7f1d5c13f970 2024-03-29T11:43:37.599 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.592+0000 7f1d646d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:37.599 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.592+0000 7f1d646d9640 10 monclient(hunting): authenticate will time out at 1376.880163s 2024-03-29T11:43:37.600 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.592+0000 7f1d62c76640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f1d5c142800 0x7f1d5c14da10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37776/0 (socket says 172.21.15.72:37776) 2024-03-29T11:43:37.600 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.592+0000 7f1d62c76640 1 -- 172.21.15.72:0/1597309841 learned_addr learned my addr 172.21.15.72:0/1597309841 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:43:37.600 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.593+0000 7f1d61473640 1 -- 172.21.15.72:0/1597309841 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 181228243 0 0) 0x7f1d5c0afe00 con 0x7f1d5c142800 2024-03-29T11:43:37.600 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.593+0000 7f1d61473640 10 monclient(hunting): _negotiate 2024-03-29T11:43:37.600 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.593+0000 7f1d61473640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:37.600 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.593+0000 7f1d61473640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:37.600 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.593+0000 7f1d61473640 10 monclient(hunting): my global_id is 4533 2024-03-29T11:43:37.600 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.593+0000 7f1d61473640 1 -- 172.21.15.72:0/1597309841 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d44003520 con 0x7f1d5c142800 2024-03-29T11:43:37.600 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.593+0000 7f1d61473640 1 -- 172.21.15.72:0/1597309841 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3803469236 0 0) 0x7f1d44003520 con 0x7f1d5c142800 2024-03-29T11:43:37.600 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.593+0000 7f1d61473640 1 -- 172.21.15.72:0/1597309841 >> v1:172.21.15.72:6790/0 conn(0x7f1d5c13f970 legacy=0x7f1d5c1420f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:43:37.604 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.597+0000 7f1d61473640 1 -- 172.21.15.72:0/1597309841 >> v1:172.21.15.89:6789/0 conn(0x7f1d5c0ae250 legacy=0x7f1d5c13f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:37.604 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.597+0000 7f1d61473640 10 monclient: _finish_hunting 0 2024-03-29T11:43:37.604 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.597+0000 7f1d61473640 1 monclient: found mon.noname-a 2024-03-29T11:43:37.604 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.597+0000 7f1d61473640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:37.604 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.597+0000 7f1d61473640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.72:6789/0 2024-03-29T11:43:37.605 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.597+0000 7f1d61473640 1 -- 172.21.15.72:0/1597309841 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d5c0aab80 con 0x7f1d5c142800 2024-03-29T11:43:37.605 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.597+0000 7f1d61473640 10 monclient: _finish_auth 0 2024-03-29T11:43:37.605 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.597+0000 7f1d61473640 10 monclient: _check_auth_tickets 2024-03-29T11:43:37.605 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.597+0000 7f1d61473640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:37.605 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.597+0000 7f1d61473640 1 -- 172.21.15.72:0/1597309841 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f1d4c001a10 con 0x7f1d5c142800 2024-03-29T11:43:37.605 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.597+0000 7f1d61473640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:37.605 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.597+0000 7f1d61473640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:43:37.605 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.597+0000 7f1d61473640 10 monclient: dump: 2024-03-29T11:43:37.605 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:43:37.605 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:37.605 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:37.605 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:37.605 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:37.605 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:37.605 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:43:37.606 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:43:37.606 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:43:37.606 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:37.606 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.597+0000 7f1d646d9640 5 monclient: authenticate success, global_id 4533 2024-03-29T11:43:37.606 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.597+0000 7f1d646d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:37.606 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.597+0000 7f1d61473640 1 -- 172.21.15.72:0/1597309841 <== mon.0 v1:172.21.15.72:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d4c001be0 con 0x7f1d5c142800 2024-03-29T11:43:37.606 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.597+0000 7f1d61473640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:37.606 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.597+0000 7f1d61473640 1 -- 172.21.15.72:0/1597309841 <== mon.0 v1:172.21.15.72:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f1d4c002030 con 0x7f1d5c142800 2024-03-29T11:43:37.606 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.597+0000 7f1d61473640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:37.606 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.597+0000 7f1d61473640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:37.606 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.597+0000 7f1d61473640 10 monclient: dump: 2024-03-29T11:43:37.606 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:43:37.606 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:37.606 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:37.607 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:37.607 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:37.607 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:37.607 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:43:37.607 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:43:37.607 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:43:37.607 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:37.607 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.597+0000 7f1d646d9640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:37.607 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.597+0000 7f1d646d9640 4 set_mon_vals no callback set 2024-03-29T11:43:37.607 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.598+0000 7f1d646d9640 10 monclient: shutdown 2024-03-29T11:43:37.607 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.598+0000 7f1d646d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:37.607 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.598+0000 7f1d646d9640 1 -- 172.21.15.72:0/1597309841 >> v1:172.21.15.72:6789/0 conn(0x7f1d5c142800 legacy=0x7f1d5c14da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:37.607 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.599+0000 7f1d646d9640 1 -- 172.21.15.72:0/1597309841 shutdown_connections 2024-03-29T11:43:37.607 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.599+0000 7f1d646d9640 1 -- 172.21.15.72:0/1597309841 >> 172.21.15.72:0/1597309841 conn(0x7f1d5c0b35e0 msgr2=0x7f1d5c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:37.607 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.600+0000 7f1d646d9640 1 -- 172.21.15.72:0/1597309841 shutdown_connections 2024-03-29T11:43:37.607 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.600+0000 7f1d646d9640 1 -- 172.21.15.72:0/1597309841 wait complete. 2024-03-29T11:43:37.608 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.601+0000 7f1d646d9640 10 monclient: build_initial_monmap 2024-03-29T11:43:37.608 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.601+0000 7f1d646d9640 10 monclient: monmap: 2024-03-29T11:43:37.608 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:43:37.608 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:37.608 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:43:37.608 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:43:37.608 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:37.608 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:37.608 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:43:37.608 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:43:37.608 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:43:37.608 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:37.609 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.601+0000 7f1d646d9640 1 librados: starting msgr at 2024-03-29T11:43:37.609 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.601+0000 7f1d646d9640 1 librados: starting objecter 2024-03-29T11:43:37.609 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.602+0000 7f1d646d9640 1 Processor -- start 2024-03-29T11:43:37.609 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.604+0000 7f1d646d9640 1 -- start start 2024-03-29T11:43:37.609 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.604+0000 7f1d646d9640 1 librados: setting wanted keys 2024-03-29T11:43:37.609 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.604+0000 7f1d646d9640 1 librados: calling monclient init 2024-03-29T11:43:37.609 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.604+0000 7f1d646d9640 10 monclient: init 2024-03-29T11:43:37.610 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.605+0000 7f1d646d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:37.610 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.605+0000 7f1d646d9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:43:37.610 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.606+0000 7f1d646d9640 10 monclient(hunting): picked mon.noname-b con 0x7f1d5c0ae250 addr v1:172.21.15.89:6789/0 2024-03-29T11:43:37.610 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.606+0000 7f1d646d9640 10 monclient(hunting): picked mon.noname-a con 0x7f1d5c13f970 addr v1:172.21.15.72:6789/0 2024-03-29T11:43:37.621 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.606+0000 7f1d646d9640 10 monclient(hunting): picked mon.noname-c con 0x7f1d5c142800 addr v1:172.21.15.72:6790/0 2024-03-29T11:43:37.621 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.606+0000 7f1d646d9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d5c1d6190 con 0x7f1d5c13f970 2024-03-29T11:43:37.621 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.606+0000 7f1d646d9640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d5c1d6430 con 0x7f1d5c0ae250 2024-03-29T11:43:37.621 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.606+0000 7f1d646d9640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d5c1d7060 con 0x7f1d5c142800 2024-03-29T11:43:37.621 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.606+0000 7f1d646d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:37.621 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.606+0000 7f1d646d9640 10 monclient(hunting): authenticate will time out at 1376.894607s 2024-03-29T11:43:37.621 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.609+0000 7f1d61c74640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f1d5c13f970 0x7f1d5c1387e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:41040/0 (socket says 172.21.15.72:41040) 2024-03-29T11:43:37.621 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.609+0000 7f1d61c74640 1 -- 172.21.15.72:0/1117701594 learned_addr learned my addr 172.21.15.72:0/1117701594 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:43:37.621 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.609+0000 7f1d52ffd640 10 client.?.objecter ms_handle_connect 0x7f1d5c13f970 2024-03-29T11:43:37.622 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.609+0000 7f1d52ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:37.622 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.609+0000 7f1d52ffd640 1 -- 172.21.15.72:0/1117701594 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4229321840 0 0) 0x7f1d5c1d6190 con 0x7f1d5c13f970 2024-03-29T11:43:37.622 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.609+0000 7f1d52ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:37.622 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.609+0000 7f1d52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:37.622 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.609+0000 7f1d52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:37.622 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.609+0000 7f1d52ffd640 10 monclient(hunting): my global_id is 4536 2024-03-29T11:43:37.622 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.609+0000 7f1d52ffd640 1 -- 172.21.15.72:0/1117701594 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d48004eb0 con 0x7f1d5c13f970 2024-03-29T11:43:37.622 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.610+0000 7f1d52ffd640 10 client.?.objecter ms_handle_connect 0x7f1d5c142800 2024-03-29T11:43:37.622 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.610+0000 7f1d52ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:37.622 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.610+0000 7f1d52ffd640 10 client.?.objecter ms_handle_connect 0x7f1d5c0ae250 2024-03-29T11:43:37.623 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.610+0000 7f1d52ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:37.623 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.610+0000 7f1d52ffd640 1 -- 172.21.15.72:0/1117701594 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3489756823 0 0) 0x7f1d5c1d7060 con 0x7f1d5c142800 2024-03-29T11:43:37.623 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.610+0000 7f1d52ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:37.623 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.610+0000 7f1d52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:37.623 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.610+0000 7f1d52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:37.623 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.610+0000 7f1d52ffd640 10 monclient(hunting): my global_id is 4547 2024-03-29T11:43:37.623 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.611+0000 7f1d52ffd640 1 -- 172.21.15.72:0/1117701594 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d5c1d6190 con 0x7f1d5c142800 2024-03-29T11:43:37.623 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.611+0000 7f1d52ffd640 1 -- 172.21.15.72:0/1117701594 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 107178259 0 0) 0x7f1d5c1d6430 con 0x7f1d5c0ae250 2024-03-29T11:43:37.623 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.611+0000 7f1d52ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:37.623 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.611+0000 7f1d52ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:37.623 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.611+0000 7f1d52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:37.623 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.611+0000 7f1d52ffd640 10 monclient(hunting): my global_id is 4483 2024-03-29T11:43:37.623 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.611+0000 7f1d52ffd640 1 -- 172.21.15.72:0/1117701594 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d5c1d7060 con 0x7f1d5c0ae250 2024-03-29T11:43:37.623 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.611+0000 7f1d52ffd640 1 -- 172.21.15.72:0/1117701594 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2797927258 0 0) 0x7f1d48004eb0 con 0x7f1d5c13f970 2024-03-29T11:43:37.623 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.611+0000 7f1d52ffd640 1 -- 172.21.15.72:0/1117701594 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d5c1d6430 con 0x7f1d5c13f970 2024-03-29T11:43:37.624 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.611+0000 7f1d52ffd640 1 -- 172.21.15.72:0/1117701594 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1176036922 0 0) 0x7f1d5c1d6190 con 0x7f1d5c142800 2024-03-29T11:43:37.624 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.611+0000 7f1d52ffd640 1 -- 172.21.15.72:0/1117701594 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d48004eb0 con 0x7f1d5c142800 2024-03-29T11:43:37.624 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.611+0000 7f1d52ffd640 1 -- 172.21.15.72:0/1117701594 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f1d580018e0 con 0x7f1d5c13f970 2024-03-29T11:43:37.624 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.611+0000 7f1d52ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:37.624 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.611+0000 7f1d52ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:43:37.624 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.611+0000 7f1d52ffd640 10 monclient(hunting): dump: 2024-03-29T11:43:37.624 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:43:37.624 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:37.624 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:37.624 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:37.624 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:37.624 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:37.624 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:43:37.624 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:43:37.624 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:43:37.624 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:37.625 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.611+0000 7f1d52ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:37.625 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.611+0000 7f1d52ffd640 1 -- 172.21.15.72:0/1117701594 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f1d4c001ef0 con 0x7f1d5c142800 2024-03-29T11:43:37.625 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.611+0000 7f1d52ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:37.625 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.611+0000 7f1d52ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:37.625 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.611+0000 7f1d52ffd640 10 monclient(hunting): dump: 2024-03-29T11:43:37.625 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:43:37.625 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:37.625 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:37.625 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:37.625 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:37.625 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:37.625 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:43:37.625 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:43:37.625 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:43:37.625 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:37.625 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.611+0000 7f1d52ffd640 1 -- 172.21.15.72:0/1117701594 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 960109175 0 0) 0x7f1d5c1d7060 con 0x7f1d5c0ae250 2024-03-29T11:43:37.626 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.611+0000 7f1d52ffd640 1 -- 172.21.15.72:0/1117701594 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d5c1d6190 con 0x7f1d5c0ae250 2024-03-29T11:43:37.626 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.612+0000 7f1d52ffd640 1 -- 172.21.15.72:0/1117701594 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f1d54001760 con 0x7f1d5c0ae250 2024-03-29T11:43:37.626 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.612+0000 7f1d52ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:37.626 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.612+0000 7f1d52ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:37.626 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.612+0000 7f1d52ffd640 10 monclient(hunting): dump: 2024-03-29T11:43:37.626 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:43:37.626 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:37.626 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:37.626 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:37.626 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:37.626 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:37.626 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:43:37.626 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:43:37.626 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:43:37.626 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:37.627 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.612+0000 7f1d52ffd640 1 -- 172.21.15.72:0/1117701594 <== mon.0 v1:172.21.15.72:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2606818143 0 0) 0x7f1d5c1d6430 con 0x7f1d5c13f970 2024-03-29T11:43:37.627 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.612+0000 7f1d52ffd640 1 -- 172.21.15.72:0/1117701594 >> v1:172.21.15.72:6790/0 conn(0x7f1d5c142800 legacy=0x7f1d5c1de530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:37.627 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.612+0000 7f1d52ffd640 1 -- 172.21.15.72:0/1117701594 >> v1:172.21.15.89:6789/0 conn(0x7f1d5c0ae250 legacy=0x7f1d5c1d6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:37.627 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.612+0000 7f1d52ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:43:37.627 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.612+0000 7f1d52ffd640 1 monclient: found mon.a 2024-03-29T11:43:37.627 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.612+0000 7f1d52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:37.627 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.612+0000 7f1d52ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:43:37.627 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.612+0000 7f1d52ffd640 1 -- 172.21.15.72:0/1117701594 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d5c1444d0 con 0x7f1d5c13f970 2024-03-29T11:43:37.627 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.612+0000 7f1d52ffd640 10 monclient: _finish_auth 0 2024-03-29T11:43:37.627 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.612+0000 7f1d52ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:43:37.627 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.612+0000 7f1d52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:37.627 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.612+0000 7f1d646d9640 5 monclient: authenticate success, global_id 4536 2024-03-29T11:43:37.627 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.612+0000 7f1d646d9640 10 monclient: _renew_subs 2024-03-29T11:43:37.627 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.613+0000 7f1d646d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:43:37.627 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.613+0000 7f1d646d9640 1 -- 172.21.15.72:0/1117701594 --> v1:172.21.15.72:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d5c144720 con 0x7f1d5c13f970 2024-03-29T11:43:37.628 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.613+0000 7f1d646d9640 10 client.4536.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:37.628 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.613+0000 7f1d52ffd640 1 -- 172.21.15.72:0/1117701594 <== mon.0 v1:172.21.15.72:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d58001a40 con 0x7f1d5c13f970 2024-03-29T11:43:37.628 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.613+0000 7f1d52ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:37.628 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.613+0000 7f1d52ffd640 1 -- 172.21.15.72:0/1117701594 <== mon.0 v1:172.21.15.72:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f1d580037d0 con 0x7f1d5c13f970 2024-03-29T11:43:37.628 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.613+0000 7f1d646d9640 10 monclient: _renew_subs 2024-03-29T11:43:37.628 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.613+0000 7f1d646d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:43:37.628 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.613+0000 7f1d646d9640 1 -- 172.21.15.72:0/1117701594 --> v1:172.21.15.72:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d5c144b90 con 0x7f1d5c13f970 2024-03-29T11:43:37.628 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.613+0000 7f1d646d9640 1 librados: init done 2024-03-29T11:43:37.628 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.613+0000 7f1d53fff640 4 set_mon_vals no callback set 2024-03-29T11:43:37.628 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.613+0000 7f1d52ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:37.629 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.613+0000 7f1d52ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:37.629 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.613+0000 7f1d52ffd640 10 monclient: dump: 2024-03-29T11:43:37.629 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:43:37.629 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:37.629 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:37.629 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:37.629 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:37.629 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:37.629 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:43:37.629 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:43:37.629 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:43:37.629 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:37.629 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.615+0000 7f1d646d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:37.629 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.615+0000 7f1d646d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:37.629 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.615+0000 7f1d646d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:43:37.629 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.615+0000 7f1d646d9640 1 -- 172.21.15.72:0/1117701594 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d5c075f10 con 0x7f1d5c13f970 2024-03-29T11:43:37.630 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.616+0000 7f1d52ffd640 1 -- 172.21.15.72:0/1117701594 <== mon.0 v1:172.21.15.72:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f1d58003930 con 0x7f1d5c13f970 2024-03-29T11:43:37.630 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.617+0000 7f1d52ffd640 1 -- 172.21.15.72:0/1117701594 <== mon.0 v1:172.21.15.72:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f1d58054780 con 0x7f1d5c13f970 2024-03-29T11:43:37.630 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.617+0000 7f1d52ffd640 10 client.4536.objecter ms_dispatch 0x7f1d5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:37.630 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.617+0000 7f1d52ffd640 3 client.4536.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:37.630 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.617+0000 7f1d52ffd640 3 client.4536.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:37.630 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.617+0000 7f1d52ffd640 20 client.4536.objecter dump_active .. 0 homeless 2024-03-29T11:43:37.630 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.617+0000 7f1d62475640 10 monclient: get_auth_request con 0x7f1d4806e060 auth_method 0 2024-03-29T11:43:37.630 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.618+0000 7f1d52ffd640 10 client.4536.objecter ms_handle_connect 0x7f1d4806e060 2024-03-29T11:43:37.631 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.627+0000 7f1d52ffd640 1 -- 172.21.15.72:0/1117701594 <== mon.0 v1:172.21.15.72:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1d58050a20 con 0x7f1d5c13f970 2024-03-29T11:43:37.631 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.627+0000 7f1d52ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:37.631 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.627+0000 7f1d52ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:37.949 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.945+0000 7f1d646d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:37.949 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.945+0000 7f1d646d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:37.949 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.945+0000 7f1d646d9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:43:37.949 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.945+0000 7f1d646d9640 1 -- 172.21.15.72:0/1117701594 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1d5c06ab70 con 0x7f1d5c13f970 2024-03-29T11:43:37.952 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.949+0000 7f1d52ffd640 1 -- 172.21.15.72:0/1117701594 <== mon.0 v1:172.21.15.72:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f1d58050bc0 con 0x7f1d5c13f970 2024-03-29T11:43:37.952 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.949+0000 7f1d52ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:37.952 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.949+0000 7f1d52ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:37.953 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:43:37.953 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:37.957 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.953+0000 7f1d646d9640 10 librados: watch_flush enter 2024-03-29T11:43:37.957 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.953+0000 7f1d646d9640 10 librados: watch_flush exit 2024-03-29T11:43:37.957 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.953+0000 7f1d646d9640 20 client.4536.objecter shutdown clearing up homeless session... 2024-03-29T11:43:37.957 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.953+0000 7f1d646d9640 10 client.4536.objecter successfully canceled tick 2024-03-29T11:43:37.957 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.953+0000 7f1d646d9640 1 -- 172.21.15.72:0/1117701594 >> v1:172.21.15.72:6816/1578866102 conn(0x7f1d4806e060 legacy=0x7f1d48070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:37.957 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.953+0000 7f1d646d9640 10 monclient: shutdown 2024-03-29T11:43:37.957 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.953+0000 7f1d646d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:37.958 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.953+0000 7f1d646d9640 1 -- 172.21.15.72:0/1117701594 >> v1:172.21.15.72:6789/0 conn(0x7f1d5c13f970 legacy=0x7f1d5c1387e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:37.958 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.953+0000 7f1d646d9640 1 -- 172.21.15.72:0/1117701594 shutdown_connections 2024-03-29T11:43:37.958 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.953+0000 7f1d646d9640 1 -- 172.21.15.72:0/1117701594 >> 172.21.15.72:0/1117701594 conn(0x7f1d5c0b35e0 msgr2=0x7f1d5c0b2cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:37.958 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.953+0000 7f1d646d9640 1 -- 172.21.15.72:0/1117701594 shutdown_connections 2024-03-29T11:43:37.958 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.953+0000 7f1d646d9640 1 -- 172.21.15.72:0/1117701594 wait complete. 2024-03-29T11:43:37.958 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:37.954+0000 7f1d646d9640 1 librados: shutdown 2024-03-29T11:43:37.971 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:44.778 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:44.910 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.907+0000 7fb323cf1640 10 monclient: get_monmap_and_config 2024-03-29T11:43:44.911 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.907+0000 7fb323cf1640 10 monclient: build_initial_monmap 2024-03-29T11:43:44.911 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.907+0000 7fb323cf1640 10 monclient: monmap: 2024-03-29T11:43:44.911 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:43:44.911 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:44.911 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:43:44.908206+0000 2024-03-29T11:43:44.911 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:43:44.908206+0000 2024-03-29T11:43:44.911 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:44.911 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:44.911 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:43:44.911 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:43:44.911 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:43:44.911 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:44.913 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.907+0000 7fb323cf1640 1 Processor -- start 2024-03-29T11:43:44.913 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.908+0000 7fb323cf1640 1 -- start start 2024-03-29T11:43:44.913 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.908+0000 7fb323cf1640 10 monclient: init 2024-03-29T11:43:44.913 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.908+0000 7fb323cf1640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:44.913 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.908+0000 7fb323cf1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:43:44.913 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.908+0000 7fb323cf1640 10 monclient(hunting): picked mon.noname-a con 0x7fb31c1422b0 addr v1:172.21.15.72:6789/0 2024-03-29T11:43:44.913 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.908+0000 7fb323cf1640 10 monclient(hunting): picked mon.noname-c con 0x7fb31c142690 addr v1:172.21.15.89:6789/0 2024-03-29T11:43:44.913 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.909+0000 7fb323cf1640 10 monclient(hunting): picked mon.noname-b con 0x7fb31c0b17d0 addr v1:172.21.15.72:6790/0 2024-03-29T11:43:44.913 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.909+0000 7fb32228e640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7fb31c0b17d0 0x7fb31c0b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:41716/0 (socket says 172.21.15.72:41716) 2024-03-29T11:43:44.913 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.909+0000 7fb32228e640 1 -- 172.21.15.72:0/2838356951 learned_addr learned my addr 172.21.15.72:0/2838356951 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:43:44.913 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.909+0000 7fb323cf1640 1 -- 172.21.15.72:0/2838356951 --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb31c142c80 con 0x7fb31c1422b0 2024-03-29T11:43:44.913 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.909+0000 7fb323cf1640 1 -- 172.21.15.72:0/2838356951 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb31c0aae40 con 0x7fb31c142690 2024-03-29T11:43:44.913 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.909+0000 7fb323cf1640 1 -- 172.21.15.72:0/2838356951 --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb31c13fbd0 con 0x7fb31c0b17d0 2024-03-29T11:43:44.913 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.909+0000 7fb323cf1640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:44.914 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.909+0000 7fb323cf1640 10 monclient(hunting): authenticate will time out at 1384.197540s 2024-03-29T11:43:44.914 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.910+0000 7fb320a8b640 1 -- 172.21.15.72:0/2838356951 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4211181146 0 0) 0x7fb31c0aae40 con 0x7fb31c142690 2024-03-29T11:43:44.914 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.910+0000 7fb320a8b640 10 monclient(hunting): _negotiate 2024-03-29T11:43:44.914 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.910+0000 7fb320a8b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:44.914 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.910+0000 7fb320a8b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:44.914 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.910+0000 7fb320a8b640 10 monclient(hunting): my global_id is 4486 2024-03-29T11:43:44.915 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.910+0000 7fb320a8b640 1 -- 172.21.15.72:0/2838356951 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb308003520 con 0x7fb31c142690 2024-03-29T11:43:44.915 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.910+0000 7fb320a8b640 1 -- 172.21.15.72:0/2838356951 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2141947341 0 0) 0x7fb31c13fbd0 con 0x7fb31c0b17d0 2024-03-29T11:43:44.915 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.910+0000 7fb320a8b640 10 monclient(hunting): _negotiate 2024-03-29T11:43:44.915 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.910+0000 7fb320a8b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:44.915 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.910+0000 7fb320a8b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:44.915 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.910+0000 7fb320a8b640 10 monclient(hunting): my global_id is 4550 2024-03-29T11:43:44.915 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.910+0000 7fb320a8b640 1 -- 172.21.15.72:0/2838356951 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb31c0aae40 con 0x7fb31c0b17d0 2024-03-29T11:43:44.915 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.910+0000 7fb320a8b640 1 -- 172.21.15.72:0/2838356951 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2484729289 0 0) 0x7fb308003520 con 0x7fb31c142690 2024-03-29T11:43:44.915 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.910+0000 7fb320a8b640 1 -- 172.21.15.72:0/2838356951 >> v1:172.21.15.72:6790/0 conn(0x7fb31c0b17d0 legacy=0x7fb31c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:44.915 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.910+0000 7fb320a8b640 1 -- 172.21.15.72:0/2838356951 >> v1:172.21.15.72:6789/0 conn(0x7fb31c1422b0 legacy=0x7fb31c13f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:43:44.916 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.912+0000 7fb320a8b640 10 monclient: _finish_hunting 0 2024-03-29T11:43:44.916 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.912+0000 7fb320a8b640 1 monclient: found mon.noname-c 2024-03-29T11:43:44.916 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.912+0000 7fb320a8b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:44.916 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.912+0000 7fb320a8b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.89:6789/0 2024-03-29T11:43:44.916 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.912+0000 7fb320a8b640 1 -- 172.21.15.72:0/2838356951 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb31c150930 con 0x7fb31c142690 2024-03-29T11:43:44.916 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.912+0000 7fb320a8b640 10 monclient: _finish_auth 0 2024-03-29T11:43:44.916 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.912+0000 7fb320a8b640 10 monclient: _check_auth_tickets 2024-03-29T11:43:44.916 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.912+0000 7fb320a8b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:44.917 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.912+0000 7fb320a8b640 1 -- 172.21.15.72:0/2838356951 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fb30c001690 con 0x7fb31c142690 2024-03-29T11:43:44.917 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.912+0000 7fb320a8b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:44.917 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.912+0000 7fb320a8b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:44.917 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.912+0000 7fb320a8b640 10 monclient: dump: 2024-03-29T11:43:44.917 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:43:44.917 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:44.917 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:44.917 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:44.917 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:44.917 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:44.917 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:43:44.917 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:43:44.917 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:43:44.917 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:44.917 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.912+0000 7fb320a8b640 1 -- 172.21.15.72:0/2838356951 <== mon.1 v1:172.21.15.89:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb30c0017f0 con 0x7fb31c142690 2024-03-29T11:43:44.918 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.912+0000 7fb320a8b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:44.918 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.912+0000 7fb320a8b640 1 -- 172.21.15.72:0/2838356951 <== mon.1 v1:172.21.15.89:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fb30c001bc0 con 0x7fb31c142690 2024-03-29T11:43:44.918 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.912+0000 7fb320a8b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:44.918 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.912+0000 7fb320a8b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:44.918 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.912+0000 7fb320a8b640 10 monclient: dump: 2024-03-29T11:43:44.918 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:43:44.918 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:44.918 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:44.918 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:44.918 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:44.918 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:44.918 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:43:44.918 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:43:44.918 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:43:44.919 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:44.919 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.913+0000 7fb323cf1640 5 monclient: authenticate success, global_id 4486 2024-03-29T11:43:44.919 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.913+0000 7fb323cf1640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:44.919 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.913+0000 7fb323cf1640 4 set_mon_vals no callback set 2024-03-29T11:43:44.929 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.914+0000 7fb323cf1640 10 monclient: shutdown 2024-03-29T11:43:44.929 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.914+0000 7fb323cf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:44.929 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.914+0000 7fb323cf1640 1 -- 172.21.15.72:0/2838356951 >> v1:172.21.15.89:6789/0 conn(0x7fb31c142690 legacy=0x7fb31c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:44.929 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.914+0000 7fb323cf1640 1 -- 172.21.15.72:0/2838356951 shutdown_connections 2024-03-29T11:43:44.930 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.914+0000 7fb323cf1640 1 -- 172.21.15.72:0/2838356951 >> 172.21.15.72:0/2838356951 conn(0x7fb31c066a00 msgr2=0x7fb31c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:44.930 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.914+0000 7fb323cf1640 1 -- 172.21.15.72:0/2838356951 shutdown_connections 2024-03-29T11:43:44.930 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.914+0000 7fb323cf1640 1 -- 172.21.15.72:0/2838356951 wait complete. 2024-03-29T11:43:44.930 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.915+0000 7fb323cf1640 10 monclient: build_initial_monmap 2024-03-29T11:43:44.930 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.915+0000 7fb323cf1640 10 monclient: monmap: 2024-03-29T11:43:44.930 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:43:44.930 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:44.930 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:43:44.930 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:43:44.930 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:44.930 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:44.930 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:43:44.931 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:43:44.931 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:43:44.931 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:44.931 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.916+0000 7fb323cf1640 1 librados: starting msgr at 2024-03-29T11:43:44.931 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.916+0000 7fb323cf1640 1 librados: starting objecter 2024-03-29T11:43:44.931 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.916+0000 7fb323cf1640 1 Processor -- start 2024-03-29T11:43:44.931 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.916+0000 7fb323cf1640 1 -- start start 2024-03-29T11:43:44.931 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.916+0000 7fb323cf1640 1 librados: setting wanted keys 2024-03-29T11:43:44.931 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.916+0000 7fb323cf1640 1 librados: calling monclient init 2024-03-29T11:43:44.931 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.916+0000 7fb323cf1640 10 monclient: init 2024-03-29T11:43:44.931 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.917+0000 7fb323cf1640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:44.931 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.917+0000 7fb323cf1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:43:44.932 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.917+0000 7fb323cf1640 10 monclient(hunting): picked mon.noname-a con 0x7fb31c0b17d0 addr v1:172.21.15.72:6789/0 2024-03-29T11:43:44.932 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.917+0000 7fb32128c640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7fb31c1422b0 0x7fb31c0ae3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:42326/0 (socket says 172.21.15.72:42326) 2024-03-29T11:43:44.932 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.917+0000 7fb32128c640 1 -- 172.21.15.72:0/377162303 learned_addr learned my addr 172.21.15.72:0/377162303 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:43:44.932 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.918+0000 7fb323cf1640 10 monclient(hunting): picked mon.noname-c con 0x7fb31c1422b0 addr v1:172.21.15.72:6790/0 2024-03-29T11:43:44.932 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.918+0000 7fb323cf1640 10 monclient(hunting): picked mon.noname-b con 0x7fb31c142690 addr v1:172.21.15.89:6789/0 2024-03-29T11:43:44.932 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.918+0000 7fb323cf1640 1 -- 172.21.15.72:0/377162303 --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb31c0add30 con 0x7fb31c0b17d0 2024-03-29T11:43:44.932 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.918+0000 7fb3127fc640 10 client.?.objecter ms_handle_connect 0x7fb31c1422b0 2024-03-29T11:43:44.932 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.918+0000 7fb3127fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:44.932 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.918+0000 7fb323cf1640 1 -- 172.21.15.72:0/377162303 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb31c0acbe0 con 0x7fb31c142690 2024-03-29T11:43:44.932 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.918+0000 7fb323cf1640 1 -- 172.21.15.72:0/377162303 --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb31c0adfa0 con 0x7fb31c1422b0 2024-03-29T11:43:44.932 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.918+0000 7fb323cf1640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:44.933 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.918+0000 7fb323cf1640 10 monclient(hunting): authenticate will time out at 1384.206285s 2024-03-29T11:43:44.933 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.918+0000 7fb3127fc640 1 -- 172.21.15.72:0/377162303 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3507540296 0 0) 0x7fb31c0adfa0 con 0x7fb31c1422b0 2024-03-29T11:43:44.933 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.918+0000 7fb3127fc640 10 monclient(hunting): _negotiate 2024-03-29T11:43:44.933 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.918+0000 7fb3127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:44.933 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.918+0000 7fb3127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:44.933 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.918+0000 7fb3127fc640 10 monclient(hunting): my global_id is 4553 2024-03-29T11:43:44.933 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.918+0000 7fb3127fc640 1 -- 172.21.15.72:0/377162303 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb300004eb0 con 0x7fb31c1422b0 2024-03-29T11:43:44.933 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.918+0000 7fb3127fc640 10 client.?.objecter ms_handle_connect 0x7fb31c142690 2024-03-29T11:43:44.933 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.918+0000 7fb3127fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:44.933 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.918+0000 7fb3127fc640 1 -- 172.21.15.72:0/377162303 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3019580550 0 0) 0x7fb300004eb0 con 0x7fb31c1422b0 2024-03-29T11:43:44.934 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.919+0000 7fb3127fc640 1 -- 172.21.15.72:0/377162303 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb31c0adfa0 con 0x7fb31c1422b0 2024-03-29T11:43:44.934 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.919+0000 7fb3127fc640 1 -- 172.21.15.72:0/377162303 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fb30c001bc0 con 0x7fb31c1422b0 2024-03-29T11:43:44.934 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.919+0000 7fb3127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:44.934 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.919+0000 7fb3127fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:44.935 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.919+0000 7fb3127fc640 10 monclient(hunting): dump: 2024-03-29T11:43:44.935 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:43:44.935 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:44.935 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:44.935 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:44.935 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:44.935 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:44.935 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:43:44.935 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:43:44.935 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:43:44.936 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:44.936 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.919+0000 7fb3127fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:44.936 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.919+0000 7fb3127fc640 1 -- 172.21.15.72:0/377162303 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3018499356 0 0) 0x7fb31c0acbe0 con 0x7fb31c142690 2024-03-29T11:43:44.936 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.919+0000 7fb3127fc640 10 monclient(hunting): _negotiate 2024-03-29T11:43:44.936 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.919+0000 7fb3127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:44.936 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.919+0000 7fb3127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:44.936 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.919+0000 7fb3127fc640 10 monclient(hunting): my global_id is 4489 2024-03-29T11:43:44.936 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.919+0000 7fb3127fc640 1 -- 172.21.15.72:0/377162303 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb300004eb0 con 0x7fb31c142690 2024-03-29T11:43:44.936 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.919+0000 7fb3127fc640 1 -- 172.21.15.72:0/377162303 <== mon.2 v1:172.21.15.72:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 545896069 0 0) 0x7fb31c0adfa0 con 0x7fb31c1422b0 2024-03-29T11:43:44.937 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.919+0000 7fb3127fc640 1 -- 172.21.15.72:0/377162303 >> v1:172.21.15.89:6789/0 conn(0x7fb31c142690 legacy=0x7fb31c1e1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:44.937 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.919+0000 7fb3127fc640 1 -- 172.21.15.72:0/377162303 >> v1:172.21.15.72:6789/0 conn(0x7fb31c0b17d0 legacy=0x7fb31c0b62a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:43:44.937 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.919+0000 7fb3127fc640 10 monclient: _finish_hunting 0 2024-03-29T11:43:44.937 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.919+0000 7fb3127fc640 1 monclient: found mon.c 2024-03-29T11:43:44.937 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.919+0000 7fb3127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:44.937 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.919+0000 7fb3127fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:43:44.937 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.919+0000 7fb3127fc640 1 -- 172.21.15.72:0/377162303 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb31c0ab540 con 0x7fb31c1422b0 2024-03-29T11:43:44.937 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.919+0000 7fb3127fc640 10 monclient: _finish_auth 0 2024-03-29T11:43:44.937 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.919+0000 7fb3127fc640 10 monclient: _check_auth_tickets 2024-03-29T11:43:44.937 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.919+0000 7fb3127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:44.937 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.919+0000 7fb323cf1640 5 monclient: authenticate success, global_id 4553 2024-03-29T11:43:44.937 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.919+0000 7fb323cf1640 10 monclient: _renew_subs 2024-03-29T11:43:44.937 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.919+0000 7fb323cf1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:43:44.938 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.919+0000 7fb323cf1640 1 -- 172.21.15.72:0/377162303 --> v1:172.21.15.72:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb31c0ab8d0 con 0x7fb31c1422b0 2024-03-29T11:43:44.938 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.920+0000 7fb323cf1640 10 client.4553.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:44.938 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.920+0000 7fb323cf1640 10 monclient: _renew_subs 2024-03-29T11:43:44.938 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.920+0000 7fb323cf1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:43:44.938 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.920+0000 7fb323cf1640 1 -- 172.21.15.72:0/377162303 --> v1:172.21.15.72:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb31c0ac5e0 con 0x7fb31c1422b0 2024-03-29T11:43:44.938 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.920+0000 7fb323cf1640 1 librados: init done 2024-03-29T11:43:44.938 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.920+0000 7fb3127fc640 1 -- 172.21.15.72:0/377162303 <== mon.2 v1:172.21.15.72:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb30c001270 con 0x7fb31c1422b0 2024-03-29T11:43:44.938 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.920+0000 7fb3127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:44.938 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.920+0000 7fb3127fc640 1 -- 172.21.15.72:0/377162303 <== mon.2 v1:172.21.15.72:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fb30c003a40 con 0x7fb31c1422b0 2024-03-29T11:43:44.938 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.920+0000 7fb3127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:44.938 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.920+0000 7fb3127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:44.938 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.920+0000 7fb3127fc640 10 monclient: dump: 2024-03-29T11:43:44.938 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:43:44.939 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:44.939 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:44.939 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:44.939 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:44.939 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:44.939 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:43:44.939 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:43:44.939 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:43:44.939 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:44.939 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.920+0000 7fb313fff640 4 set_mon_vals no callback set 2024-03-29T11:43:44.939 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.921+0000 7fb323cf1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:44.939 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.922+0000 7fb323cf1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:44.939 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.922+0000 7fb323cf1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:43:44.939 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.922+0000 7fb323cf1640 1 -- 172.21.15.72:0/377162303 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb304005080 con 0x7fb31c1422b0 2024-03-29T11:43:44.939 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.922+0000 7fb3127fc640 1 -- 172.21.15.72:0/377162303 <== mon.2 v1:172.21.15.72:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7fb30c003ba0 con 0x7fb31c1422b0 2024-03-29T11:43:44.939 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.923+0000 7fb3127fc640 1 -- 172.21.15.72:0/377162303 <== mon.2 v1:172.21.15.72:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7fb30c084ba0 con 0x7fb31c1422b0 2024-03-29T11:43:44.940 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.923+0000 7fb3127fc640 10 client.4553.objecter ms_dispatch 0x7fb31c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:44.940 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.923+0000 7fb3127fc640 3 client.4553.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:44.940 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.923+0000 7fb3127fc640 3 client.4553.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:44.940 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.923+0000 7fb321a8d640 10 monclient: get_auth_request con 0x7fb30006dd90 auth_method 0 2024-03-29T11:43:44.940 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.923+0000 7fb3127fc640 20 client.4553.objecter dump_active .. 0 homeless 2024-03-29T11:43:44.940 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.923+0000 7fb3127fc640 10 client.4553.objecter ms_handle_connect 0x7fb30006dd90 2024-03-29T11:43:44.940 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.929+0000 7fb3127fc640 1 -- 172.21.15.72:0/377162303 <== mon.2 v1:172.21.15.72:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb30c050480 con 0x7fb31c1422b0 2024-03-29T11:43:44.940 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.929+0000 7fb3127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:44.941 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:44.929+0000 7fb3127fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:45.240 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:45.235+0000 7fb323cf1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:45.240 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:45.235+0000 7fb323cf1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:45.240 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:45.235+0000 7fb323cf1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:43:45.240 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:45.235+0000 7fb323cf1640 1 -- 172.21.15.72:0/377162303 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb304005330 con 0x7fb31c1422b0 2024-03-29T11:43:45.240 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:45.236+0000 7fb3127fc640 1 -- 172.21.15.72:0/377162303 <== mon.2 v1:172.21.15.72:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7fb30c0523b0 con 0x7fb31c1422b0 2024-03-29T11:43:45.240 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:45.236+0000 7fb3127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:45.240 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:45.236+0000 7fb3127fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:45.240 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:43:45.241 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:45.243 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:45.239+0000 7fb323cf1640 10 librados: watch_flush enter 2024-03-29T11:43:45.244 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:45.239+0000 7fb323cf1640 10 librados: watch_flush exit 2024-03-29T11:43:45.244 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:45.239+0000 7fb323cf1640 20 client.4553.objecter shutdown clearing up homeless session... 2024-03-29T11:43:45.244 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:45.239+0000 7fb323cf1640 10 client.4553.objecter successfully canceled tick 2024-03-29T11:43:45.244 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:45.239+0000 7fb323cf1640 1 -- 172.21.15.72:0/377162303 >> v1:172.21.15.72:6816/1578866102 conn(0x7fb30006dd90 legacy=0x7fb300070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:45.244 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:45.239+0000 7fb323cf1640 10 monclient: shutdown 2024-03-29T11:43:45.244 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:45.239+0000 7fb323cf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:45.244 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:45.239+0000 7fb323cf1640 1 -- 172.21.15.72:0/377162303 >> v1:172.21.15.72:6790/0 conn(0x7fb31c1422b0 legacy=0x7fb31c0ae3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:45.244 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:45.239+0000 7fb323cf1640 1 -- 172.21.15.72:0/377162303 shutdown_connections 2024-03-29T11:43:45.244 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:45.239+0000 7fb323cf1640 1 -- 172.21.15.72:0/377162303 >> 172.21.15.72:0/377162303 conn(0x7fb31c066a00 msgr2=0x7fb31c13c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:45.244 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:45.240+0000 7fb323cf1640 1 -- 172.21.15.72:0/377162303 shutdown_connections 2024-03-29T11:43:45.244 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:45.240+0000 7fb323cf1640 1 -- 172.21.15.72:0/377162303 wait complete. 2024-03-29T11:43:45.244 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:45.240+0000 7fb323cf1640 1 librados: shutdown 2024-03-29T11:43:45.258 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:52.066 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:52.203 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.199+0000 7faad80b3640 10 monclient: get_monmap_and_config 2024-03-29T11:43:52.203 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.199+0000 7faad80b3640 10 monclient: build_initial_monmap 2024-03-29T11:43:52.204 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.199+0000 7faad80b3640 10 monclient: monmap: 2024-03-29T11:43:52.204 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:43:52.204 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:52.204 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:43:52.200746+0000 2024-03-29T11:43:52.204 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:43:52.200746+0000 2024-03-29T11:43:52.204 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:52.204 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:52.204 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:43:52.204 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:43:52.204 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:43:52.204 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:52.207 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.200+0000 7faad80b3640 1 Processor -- start 2024-03-29T11:43:52.207 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.200+0000 7faad80b3640 1 -- start start 2024-03-29T11:43:52.207 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.201+0000 7faad80b3640 10 monclient: init 2024-03-29T11:43:52.207 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.201+0000 7faad80b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:52.207 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.201+0000 7faad80b3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:43:52.207 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.201+0000 7faad80b3640 10 monclient(hunting): picked mon.noname-a con 0x7faad01422b0 addr v1:172.21.15.72:6789/0 2024-03-29T11:43:52.207 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.201+0000 7faad80b3640 10 monclient(hunting): picked mon.noname-b con 0x7faad0142690 addr v1:172.21.15.72:6790/0 2024-03-29T11:43:52.207 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.202+0000 7faad80b3640 10 monclient(hunting): picked mon.noname-c con 0x7faad00b17d0 addr v1:172.21.15.89:6789/0 2024-03-29T11:43:52.207 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.202+0000 7faad5e4f640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7faad01422b0 0x7faad013f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:34158/0 (socket says 172.21.15.72:34158) 2024-03-29T11:43:52.207 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.202+0000 7faad5e4f640 1 -- 172.21.15.72:0/3636149678 learned_addr learned my addr 172.21.15.72:0/3636149678 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:43:52.207 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.202+0000 7faad80b3640 1 -- 172.21.15.72:0/3636149678 --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faad0142c80 con 0x7faad01422b0 2024-03-29T11:43:52.208 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.202+0000 7faad80b3640 1 -- 172.21.15.72:0/3636149678 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faad00aae40 con 0x7faad00b17d0 2024-03-29T11:43:52.208 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.202+0000 7faad80b3640 1 -- 172.21.15.72:0/3636149678 --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faad013fbd0 con 0x7faad0142690 2024-03-29T11:43:52.208 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.202+0000 7faad80b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:52.208 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.202+0000 7faad80b3640 10 monclient(hunting): authenticate will time out at 1391.490524s 2024-03-29T11:43:52.208 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.202+0000 7faad4e4d640 1 -- 172.21.15.72:0/3636149678 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2675239916 0 0) 0x7faad0142c80 con 0x7faad01422b0 2024-03-29T11:43:52.208 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.202+0000 7faad4e4d640 10 monclient(hunting): _negotiate 2024-03-29T11:43:52.208 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.202+0000 7faad4e4d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:52.208 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.202+0000 7faad4e4d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:52.208 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.202+0000 7faad4e4d640 10 monclient(hunting): my global_id is 4539 2024-03-29T11:43:52.208 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.202+0000 7faad4e4d640 1 -- 172.21.15.72:0/3636149678 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faabc003520 con 0x7faad01422b0 2024-03-29T11:43:52.208 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.202+0000 7faad4e4d640 1 -- 172.21.15.72:0/3636149678 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2669570548 0 0) 0x7faabc003520 con 0x7faad01422b0 2024-03-29T11:43:52.208 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.203+0000 7faad4e4d640 1 -- 172.21.15.72:0/3636149678 >> v1:172.21.15.72:6790/0 conn(0x7faad0142690 legacy=0x7faad00b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:43:52.211 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.205+0000 7faad4e4d640 1 -- 172.21.15.72:0/3636149678 >> v1:172.21.15.89:6789/0 conn(0x7faad00b17d0 legacy=0x7faad00b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:52.211 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.205+0000 7faad4e4d640 10 monclient: _finish_hunting 0 2024-03-29T11:43:52.211 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.205+0000 7faad4e4d640 1 monclient: found mon.noname-a 2024-03-29T11:43:52.211 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.205+0000 7faad4e4d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:52.212 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.205+0000 7faad4e4d640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.72:6789/0 2024-03-29T11:43:52.212 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.205+0000 7faad4e4d640 1 -- 172.21.15.72:0/3636149678 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad0150930 con 0x7faad01422b0 2024-03-29T11:43:52.212 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.205+0000 7faad4e4d640 10 monclient: _finish_auth 0 2024-03-29T11:43:52.212 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.205+0000 7faad4e4d640 10 monclient: _check_auth_tickets 2024-03-29T11:43:52.212 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.205+0000 7faad4e4d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:52.212 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.205+0000 7faad80b3640 5 monclient: authenticate success, global_id 4539 2024-03-29T11:43:52.212 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.205+0000 7faad80b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:52.212 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.205+0000 7faad4e4d640 1 -- 172.21.15.72:0/3636149678 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7faacc001690 con 0x7faad01422b0 2024-03-29T11:43:52.212 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.205+0000 7faad4e4d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:52.213 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.205+0000 7faad4e4d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:43:52.213 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.205+0000 7faad4e4d640 10 monclient: dump: 2024-03-29T11:43:52.213 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:43:52.213 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:52.213 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:52.213 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:52.213 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:52.213 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:52.213 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:43:52.213 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:43:52.213 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:43:52.213 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:52.214 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.205+0000 7faad4e4d640 1 -- 172.21.15.72:0/3636149678 <== mon.0 v1:172.21.15.72:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faacc001930 con 0x7faad01422b0 2024-03-29T11:43:52.214 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.205+0000 7faad4e4d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:52.214 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.205+0000 7faad4e4d640 1 -- 172.21.15.72:0/3636149678 <== mon.0 v1:172.21.15.72:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7faacc001d00 con 0x7faad01422b0 2024-03-29T11:43:52.214 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.205+0000 7faad4e4d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:52.214 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.205+0000 7faad4e4d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:52.214 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.205+0000 7faad4e4d640 10 monclient: dump: 2024-03-29T11:43:52.214 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:43:52.214 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:52.214 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:52.214 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:52.214 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:52.214 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:52.214 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:43:52.214 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:43:52.215 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:43:52.215 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:52.215 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.205+0000 7faad80b3640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:52.215 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.205+0000 7faad80b3640 4 set_mon_vals no callback set 2024-03-29T11:43:52.215 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.206+0000 7faad80b3640 10 monclient: shutdown 2024-03-29T11:43:52.215 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.206+0000 7faad80b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:52.215 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.206+0000 7faad80b3640 1 -- 172.21.15.72:0/3636149678 >> v1:172.21.15.72:6789/0 conn(0x7faad01422b0 legacy=0x7faad013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:52.215 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.206+0000 7faad80b3640 1 -- 172.21.15.72:0/3636149678 shutdown_connections 2024-03-29T11:43:52.215 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.206+0000 7faad80b3640 1 -- 172.21.15.72:0/3636149678 >> 172.21.15.72:0/3636149678 conn(0x7faad0066a00 msgr2=0x7faad0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:52.215 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.207+0000 7faad80b3640 1 -- 172.21.15.72:0/3636149678 shutdown_connections 2024-03-29T11:43:52.215 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.207+0000 7faad80b3640 1 -- 172.21.15.72:0/3636149678 wait complete. 2024-03-29T11:43:52.221 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.208+0000 7faad80b3640 10 monclient: build_initial_monmap 2024-03-29T11:43:52.221 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.208+0000 7faad80b3640 10 monclient: monmap: 2024-03-29T11:43:52.221 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:43:52.221 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:52.221 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:43:52.221 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:43:52.221 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:52.221 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:52.221 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:43:52.221 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:43:52.221 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:43:52.221 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:52.221 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.209+0000 7faad80b3640 1 librados: starting msgr at 2024-03-29T11:43:52.222 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.209+0000 7faad80b3640 1 librados: starting objecter 2024-03-29T11:43:52.222 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.209+0000 7faad80b3640 1 Processor -- start 2024-03-29T11:43:52.222 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.209+0000 7faad80b3640 1 -- start start 2024-03-29T11:43:52.222 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.209+0000 7faad80b3640 1 librados: setting wanted keys 2024-03-29T11:43:52.222 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.209+0000 7faad80b3640 1 librados: calling monclient init 2024-03-29T11:43:52.222 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.209+0000 7faad80b3640 10 monclient: init 2024-03-29T11:43:52.222 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.210+0000 7faad80b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:52.222 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.210+0000 7faad80b3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:43:52.222 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.210+0000 7faad80b3640 10 monclient(hunting): picked mon.noname-b con 0x7faad00b17d0 addr v1:172.21.15.89:6789/0 2024-03-29T11:43:52.222 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.211+0000 7faad5e4f640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7faad00b17d0 0x7faad00ae1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:51260/0 (socket says 172.21.15.72:51260) 2024-03-29T11:43:52.222 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.212+0000 7faad80b3640 10 monclient(hunting): picked mon.noname-c con 0x7faad01422b0 addr v1:172.21.15.72:6790/0 2024-03-29T11:43:52.222 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.212+0000 7faad5e4f640 1 -- 172.21.15.72:0/798867822 learned_addr learned my addr 172.21.15.72:0/798867822 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:43:52.222 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.212+0000 7faac67fc640 10 client.?.objecter ms_handle_connect 0x7faad00b17d0 2024-03-29T11:43:52.222 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.212+0000 7faac67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:52.222 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.212+0000 7faad80b3640 10 monclient(hunting): picked mon.noname-a con 0x7faad0142690 addr v1:172.21.15.72:6789/0 2024-03-29T11:43:52.223 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.212+0000 7faac67fc640 10 client.?.objecter ms_handle_connect 0x7faad0142690 2024-03-29T11:43:52.223 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.212+0000 7faac67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:52.223 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.212+0000 7faad80b3640 1 -- 172.21.15.72:0/798867822 --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faad00abd10 con 0x7faad0142690 2024-03-29T11:43:52.223 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.212+0000 7faad80b3640 1 -- 172.21.15.72:0/798867822 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faad00ad7c0 con 0x7faad00b17d0 2024-03-29T11:43:52.223 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.213+0000 7faac67fc640 1 -- 172.21.15.72:0/798867822 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 333889313 0 0) 0x7faad00abd10 con 0x7faad0142690 2024-03-29T11:43:52.223 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.213+0000 7faad80b3640 1 -- 172.21.15.72:0/798867822 --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faad00adaa0 con 0x7faad01422b0 2024-03-29T11:43:52.223 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.213+0000 7faad80b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:52.223 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.213+0000 7faad80b3640 10 monclient(hunting): authenticate will time out at 1391.501292s 2024-03-29T11:43:52.223 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.213+0000 7faac67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:43:52.223 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.213+0000 7faac67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:52.223 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.213+0000 7faac67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:52.223 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.213+0000 7faac67fc640 10 monclient(hunting): my global_id is 4542 2024-03-29T11:43:52.223 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.213+0000 7faac67fc640 1 -- 172.21.15.72:0/798867822 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faab4004eb0 con 0x7faad0142690 2024-03-29T11:43:52.223 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.213+0000 7faac67fc640 1 -- 172.21.15.72:0/798867822 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4193550896 0 0) 0x7faad00ad7c0 con 0x7faad00b17d0 2024-03-29T11:43:52.223 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.213+0000 7faac67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:43:52.224 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.213+0000 7faac67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:52.224 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.213+0000 7faac67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:52.224 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.213+0000 7faac67fc640 10 monclient(hunting): my global_id is 4495 2024-03-29T11:43:52.224 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.213+0000 7faac67fc640 1 -- 172.21.15.72:0/798867822 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faad00abd10 con 0x7faad00b17d0 2024-03-29T11:43:52.224 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.213+0000 7faac67fc640 1 -- 172.21.15.72:0/798867822 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 737803102 0 0) 0x7faab4004eb0 con 0x7faad0142690 2024-03-29T11:43:52.224 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.213+0000 7faac67fc640 1 -- 172.21.15.72:0/798867822 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faad00ad7c0 con 0x7faad0142690 2024-03-29T11:43:52.224 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.213+0000 7faac67fc640 1 -- 172.21.15.72:0/798867822 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7faac8001c30 con 0x7faad0142690 2024-03-29T11:43:52.224 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.213+0000 7faac67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:52.224 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.213+0000 7faac67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:43:52.224 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.213+0000 7faac67fc640 10 monclient(hunting): dump: 2024-03-29T11:43:52.224 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:43:52.224 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:52.224 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:52.224 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:52.224 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:52.224 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:52.225 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:43:52.225 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:43:52.225 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:43:52.225 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:52.225 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.213+0000 7faac67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:52.225 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.213+0000 7faac67fc640 1 -- 172.21.15.72:0/798867822 <== mon.0 v1:172.21.15.72:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2964370601 0 0) 0x7faad00ad7c0 con 0x7faad0142690 2024-03-29T11:43:52.225 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.213+0000 7faac67fc640 1 -- 172.21.15.72:0/798867822 >> v1:172.21.15.72:6790/0 conn(0x7faad01422b0 legacy=0x7faad00ac430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:43:52.225 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.214+0000 7faac67fc640 1 -- 172.21.15.72:0/798867822 >> v1:172.21.15.89:6789/0 conn(0x7faad00b17d0 legacy=0x7faad00ae1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:52.225 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.214+0000 7faac67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:43:52.225 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.214+0000 7faac67fc640 1 monclient: found mon.a 2024-03-29T11:43:52.225 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.214+0000 7faac67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:52.225 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.214+0000 7faac67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:43:52.225 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.214+0000 7faac67fc640 1 -- 172.21.15.72:0/798867822 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad00add80 con 0x7faad0142690 2024-03-29T11:43:52.225 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.214+0000 7faac67fc640 10 monclient: _finish_auth 0 2024-03-29T11:43:52.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.214+0000 7faac67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:43:52.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.214+0000 7faac67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:52.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.214+0000 7faad80b3640 5 monclient: authenticate success, global_id 4542 2024-03-29T11:43:52.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.214+0000 7faad80b3640 10 monclient: _renew_subs 2024-03-29T11:43:52.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.214+0000 7faad80b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:43:52.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.214+0000 7faad80b3640 1 -- 172.21.15.72:0/798867822 --> v1:172.21.15.72:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faad00acb40 con 0x7faad0142690 2024-03-29T11:43:52.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.214+0000 7faad80b3640 10 client.4542.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:52.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.214+0000 7faad80b3640 10 monclient: _renew_subs 2024-03-29T11:43:52.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.214+0000 7faad80b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:43:52.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.214+0000 7faad80b3640 1 -- 172.21.15.72:0/798867822 --> v1:172.21.15.72:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faad00b67a0 con 0x7faad0142690 2024-03-29T11:43:52.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.214+0000 7faad80b3640 1 librados: init done 2024-03-29T11:43:52.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.214+0000 7faac67fc640 1 -- 172.21.15.72:0/798867822 <== mon.0 v1:172.21.15.72:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faac8002080 con 0x7faad0142690 2024-03-29T11:43:52.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.214+0000 7faac67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:52.227 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.214+0000 7faac67fc640 1 -- 172.21.15.72:0/798867822 <== mon.0 v1:172.21.15.72:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7faac8003b30 con 0x7faad0142690 2024-03-29T11:43:52.227 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.214+0000 7faac67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:52.227 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.214+0000 7faac67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:52.227 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.214+0000 7faac67fc640 10 monclient: dump: 2024-03-29T11:43:52.227 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:43:52.227 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:52.227 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:52.227 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:52.227 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:52.227 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:52.227 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:43:52.227 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:43:52.227 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:43:52.227 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:52.227 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.214+0000 7faac7fff640 4 set_mon_vals no callback set 2024-03-29T11:43:52.227 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.216+0000 7faad80b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:52.228 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.216+0000 7faad80b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:52.228 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.216+0000 7faad80b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:43:52.228 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.216+0000 7faad80b3640 1 -- 172.21.15.72:0/798867822 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faab8005080 con 0x7faad0142690 2024-03-29T11:43:52.228 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.217+0000 7faac67fc640 1 -- 172.21.15.72:0/798867822 <== mon.0 v1:172.21.15.72:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7faac8019b00 con 0x7faad0142690 2024-03-29T11:43:52.229 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.217+0000 7faac67fc640 1 -- 172.21.15.72:0/798867822 <== mon.0 v1:172.21.15.72:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7faac8083ab0 con 0x7faad0142690 2024-03-29T11:43:52.229 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.217+0000 7faac67fc640 10 client.4542.objecter ms_dispatch 0x7faad0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:52.229 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.217+0000 7faac67fc640 3 client.4542.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:52.229 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.217+0000 7faac67fc640 3 client.4542.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:52.229 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.218+0000 7faac67fc640 20 client.4542.objecter dump_active .. 0 homeless 2024-03-29T11:43:52.229 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.218+0000 7faad5e4f640 10 monclient: get_auth_request con 0x7faab406dd30 auth_method 0 2024-03-29T11:43:52.229 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.218+0000 7faac67fc640 10 client.4542.objecter ms_handle_connect 0x7faab406dd30 2024-03-29T11:43:52.229 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.225+0000 7faac67fc640 1 -- 172.21.15.72:0/798867822 <== mon.0 v1:172.21.15.72:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faac804f1e0 con 0x7faad0142690 2024-03-29T11:43:52.229 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.225+0000 7faac67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:52.229 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.225+0000 7faac67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:52.538 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.534+0000 7faad80b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:52.538 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.534+0000 7faad80b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:52.538 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.534+0000 7faad80b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:43:52.539 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.534+0000 7faad80b3640 1 -- 172.21.15.72:0/798867822 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faab8005330 con 0x7faad0142690 2024-03-29T11:43:52.539 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.535+0000 7faac67fc640 1 -- 172.21.15.72:0/798867822 <== mon.0 v1:172.21.15.72:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7faac80512e0 con 0x7faad0142690 2024-03-29T11:43:52.539 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.535+0000 7faac67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:52.539 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.535+0000 7faac67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:52.539 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:43:52.539 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:52.543 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.538+0000 7faad80b3640 10 librados: watch_flush enter 2024-03-29T11:43:52.543 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.538+0000 7faad80b3640 10 librados: watch_flush exit 2024-03-29T11:43:52.543 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.538+0000 7faad80b3640 20 client.4542.objecter shutdown clearing up homeless session... 2024-03-29T11:43:52.543 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.538+0000 7faad80b3640 10 client.4542.objecter successfully canceled tick 2024-03-29T11:43:52.543 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.538+0000 7faad80b3640 1 -- 172.21.15.72:0/798867822 >> v1:172.21.15.72:6816/1578866102 conn(0x7faab406dd30 legacy=0x7faab40701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:52.543 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.538+0000 7faad80b3640 10 monclient: shutdown 2024-03-29T11:43:52.543 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.538+0000 7faad80b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:52.543 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.538+0000 7faad80b3640 1 -- 172.21.15.72:0/798867822 >> v1:172.21.15.72:6789/0 conn(0x7faad0142690 legacy=0x7faad01e1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:52.543 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.539+0000 7faad80b3640 1 -- 172.21.15.72:0/798867822 shutdown_connections 2024-03-29T11:43:52.543 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.539+0000 7faad80b3640 1 -- 172.21.15.72:0/798867822 >> 172.21.15.72:0/798867822 conn(0x7faad0066a00 msgr2=0x7faad013c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:52.543 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.539+0000 7faad80b3640 1 -- 172.21.15.72:0/798867822 shutdown_connections 2024-03-29T11:43:52.543 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.539+0000 7faad80b3640 1 -- 172.21.15.72:0/798867822 wait complete. 2024-03-29T11:43:52.544 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:52.539+0000 7faad80b3640 1 librados: shutdown 2024-03-29T11:43:52.557 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:59.365 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:59.512 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.508+0000 7efc65523640 10 monclient: get_monmap_and_config 2024-03-29T11:43:59.512 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.508+0000 7efc65523640 10 monclient: build_initial_monmap 2024-03-29T11:43:59.512 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.508+0000 7efc65523640 10 monclient: monmap: 2024-03-29T11:43:59.512 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:43:59.512 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:59.512 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:43:59.509271+0000 2024-03-29T11:43:59.512 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:43:59.509271+0000 2024-03-29T11:43:59.512 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:59.512 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:59.512 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:43:59.513 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:43:59.513 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:43:59.513 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:59.515 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.509+0000 7efc65523640 1 Processor -- start 2024-03-29T11:43:59.516 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.509+0000 7efc65523640 1 -- start start 2024-03-29T11:43:59.516 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.509+0000 7efc65523640 10 monclient: init 2024-03-29T11:43:59.516 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.510+0000 7efc65523640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:59.516 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.510+0000 7efc65523640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:43:59.516 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.510+0000 7efc65523640 10 monclient(hunting): picked mon.noname-b con 0x7efc601457f0 addr v1:172.21.15.72:6790/0 2024-03-29T11:43:59.516 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.510+0000 7efc65523640 10 monclient(hunting): picked mon.noname-a con 0x7efc6013f650 addr v1:172.21.15.72:6789/0 2024-03-29T11:43:59.516 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.510+0000 7efc65523640 10 monclient(hunting): picked mon.noname-c con 0x7efc600a9d70 addr v1:172.21.15.89:6789/0 2024-03-29T11:43:59.516 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.510+0000 7efc65523640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc600aaef0 con 0x7efc6013f650 2024-03-29T11:43:59.516 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.510+0000 7efc65523640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc60145bd0 con 0x7efc600a9d70 2024-03-29T11:43:59.516 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.510+0000 7efc65523640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc60145eb0 con 0x7efc601457f0 2024-03-29T11:43:59.516 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.510+0000 7efc65523640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:59.516 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.510+0000 7efc65523640 10 monclient(hunting): authenticate will time out at 1398.798142s 2024-03-29T11:43:59.516 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.510+0000 7efc5effd640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7efc601457f0 0x7efc6013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:58238/0 (socket says 172.21.15.72:58238) 2024-03-29T11:43:59.516 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.510+0000 7efc5effd640 1 -- 172.21.15.72:0/3127514624 learned_addr learned my addr 172.21.15.72:0/3127514624 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:43:59.516 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.510+0000 7efc5e7fc640 1 -- 172.21.15.72:0/3127514624 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 875544262 0 0) 0x7efc60145eb0 con 0x7efc601457f0 2024-03-29T11:43:59.517 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.510+0000 7efc5e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:43:59.517 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.510+0000 7efc5e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:59.517 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.510+0000 7efc5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:59.517 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.510+0000 7efc5e7fc640 10 monclient(hunting): my global_id is 4556 2024-03-29T11:43:59.517 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.510+0000 7efc5e7fc640 1 -- 172.21.15.72:0/3127514624 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc4c003520 con 0x7efc601457f0 2024-03-29T11:43:59.517 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.511+0000 7efc5e7fc640 1 -- 172.21.15.72:0/3127514624 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1116841008 0 0) 0x7efc4c003520 con 0x7efc601457f0 2024-03-29T11:43:59.517 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.511+0000 7efc5e7fc640 1 -- 172.21.15.72:0/3127514624 >> v1:172.21.15.89:6789/0 conn(0x7efc600a9d70 legacy=0x7efc6014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:43:59.517 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.511+0000 7efc5e7fc640 1 -- 172.21.15.72:0/3127514624 >> v1:172.21.15.72:6789/0 conn(0x7efc6013f650 legacy=0x7efc600b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:43:59.519 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.513+0000 7efc5e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:43:59.519 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.513+0000 7efc5e7fc640 1 monclient: found mon.noname-b 2024-03-29T11:43:59.519 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.513+0000 7efc5e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:59.519 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.513+0000 7efc5e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.72:6790/0 2024-03-29T11:43:59.519 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.513+0000 7efc5e7fc640 1 -- 172.21.15.72:0/3127514624 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc60146340 con 0x7efc601457f0 2024-03-29T11:43:59.520 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.513+0000 7efc5e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:43:59.520 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.513+0000 7efc5e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:43:59.520 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.513+0000 7efc5e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:59.520 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.513+0000 7efc5e7fc640 1 -- 172.21.15.72:0/3127514624 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7efc50001690 con 0x7efc601457f0 2024-03-29T11:43:59.520 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.513+0000 7efc5e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:59.520 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.513+0000 7efc5e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:59.520 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.513+0000 7efc5e7fc640 10 monclient: dump: 2024-03-29T11:43:59.520 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:43:59.520 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:59.520 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:59.520 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:59.520 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:59.520 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:59.520 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:43:59.520 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:43:59.521 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:43:59.521 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:59.521 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.513+0000 7efc65523640 5 monclient: authenticate success, global_id 4556 2024-03-29T11:43:59.521 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.513+0000 7efc65523640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:59.521 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.514+0000 7efc5e7fc640 1 -- 172.21.15.72:0/3127514624 <== mon.2 v1:172.21.15.72:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc500018f0 con 0x7efc601457f0 2024-03-29T11:43:59.521 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.514+0000 7efc5e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:59.521 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.514+0000 7efc5e7fc640 1 -- 172.21.15.72:0/3127514624 <== mon.2 v1:172.21.15.72:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7efc50001d40 con 0x7efc601457f0 2024-03-29T11:43:59.521 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.514+0000 7efc5e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:59.521 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.514+0000 7efc5e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:59.521 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.514+0000 7efc5e7fc640 10 monclient: dump: 2024-03-29T11:43:59.521 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:43:59.521 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:59.521 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:59.521 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:59.521 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:59.522 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:59.522 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:43:59.522 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:43:59.522 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:43:59.522 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:59.522 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.514+0000 7efc65523640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:59.522 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.514+0000 7efc65523640 4 set_mon_vals no callback set 2024-03-29T11:43:59.522 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.515+0000 7efc65523640 10 monclient: shutdown 2024-03-29T11:43:59.522 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.515+0000 7efc65523640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:59.523 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.515+0000 7efc65523640 1 -- 172.21.15.72:0/3127514624 >> v1:172.21.15.72:6790/0 conn(0x7efc601457f0 legacy=0x7efc6013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:59.523 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.515+0000 7efc65523640 1 -- 172.21.15.72:0/3127514624 shutdown_connections 2024-03-29T11:43:59.523 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.515+0000 7efc65523640 1 -- 172.21.15.72:0/3127514624 >> 172.21.15.72:0/3127514624 conn(0x7efc60066a00 msgr2=0x7efc60066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:59.523 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.516+0000 7efc65523640 1 -- 172.21.15.72:0/3127514624 shutdown_connections 2024-03-29T11:43:59.530 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.516+0000 7efc65523640 1 -- 172.21.15.72:0/3127514624 wait complete. 2024-03-29T11:43:59.530 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.517+0000 7efc65523640 10 monclient: build_initial_monmap 2024-03-29T11:43:59.530 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.517+0000 7efc65523640 10 monclient: monmap: 2024-03-29T11:43:59.530 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:43:59.530 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:59.530 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:43:59.530 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:43:59.530 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:59.530 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:59.530 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:43:59.531 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:43:59.531 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:43:59.531 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:59.531 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.517+0000 7efc65523640 1 librados: starting msgr at 2024-03-29T11:43:59.531 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.517+0000 7efc65523640 1 librados: starting objecter 2024-03-29T11:43:59.531 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.517+0000 7efc65523640 1 Processor -- start 2024-03-29T11:43:59.531 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.517+0000 7efc65523640 1 -- start start 2024-03-29T11:43:59.531 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.517+0000 7efc65523640 1 librados: setting wanted keys 2024-03-29T11:43:59.531 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.517+0000 7efc65523640 1 librados: calling monclient init 2024-03-29T11:43:59.531 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.517+0000 7efc65523640 10 monclient: init 2024-03-29T11:43:59.531 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.518+0000 7efc65523640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:59.531 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.518+0000 7efc65523640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:43:59.531 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.518+0000 7efc65523640 10 monclient(hunting): picked mon.noname-a con 0x7efc600a9d70 addr v1:172.21.15.72:6789/0 2024-03-29T11:43:59.531 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.518+0000 7efc65523640 10 monclient(hunting): picked mon.noname-b con 0x7efc6013f650 addr v1:172.21.15.89:6789/0 2024-03-29T11:43:59.531 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.518+0000 7efc65523640 10 monclient(hunting): picked mon.noname-c con 0x7efc601d3d40 addr v1:172.21.15.72:6790/0 2024-03-29T11:43:59.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.518+0000 7efc65523640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc601d4190 con 0x7efc600a9d70 2024-03-29T11:43:59.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.518+0000 7efc65523640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc601d8150 con 0x7efc6013f650 2024-03-29T11:43:59.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.518+0000 7efc65523640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc601350f0 con 0x7efc601d3d40 2024-03-29T11:43:59.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.518+0000 7efc65523640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:59.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.518+0000 7efc65523640 10 monclient(hunting): authenticate will time out at 1398.806647s 2024-03-29T11:43:59.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.518+0000 7efc5f7fe640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7efc601d3d40 0x7efc60133400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:60600/0 (socket says 172.21.15.72:60600) 2024-03-29T11:43:59.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.518+0000 7efc5f7fe640 1 -- 172.21.15.72:0/102033489 learned_addr learned my addr 172.21.15.72:0/102033489 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:43:59.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.519+0000 7efc577fe640 10 client.?.objecter ms_handle_connect 0x7efc601d3d40 2024-03-29T11:43:59.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.519+0000 7efc577fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:59.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.519+0000 7efc577fe640 1 -- 172.21.15.72:0/102033489 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2072572763 0 0) 0x7efc601350f0 con 0x7efc601d3d40 2024-03-29T11:43:59.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.519+0000 7efc577fe640 10 monclient(hunting): _negotiate 2024-03-29T11:43:59.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.519+0000 7efc577fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:59.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.519+0000 7efc577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:59.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.519+0000 7efc577fe640 10 monclient(hunting): my global_id is 4559 2024-03-29T11:43:59.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.519+0000 7efc577fe640 1 -- 172.21.15.72:0/102033489 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc40004eb0 con 0x7efc601d3d40 2024-03-29T11:43:59.533 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.519+0000 7efc577fe640 1 -- 172.21.15.72:0/102033489 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3446967203 0 0) 0x7efc40004eb0 con 0x7efc601d3d40 2024-03-29T11:43:59.533 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.519+0000 7efc577fe640 1 -- 172.21.15.72:0/102033489 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc601350f0 con 0x7efc601d3d40 2024-03-29T11:43:59.533 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.519+0000 7efc577fe640 1 -- 172.21.15.72:0/102033489 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7efc58001c60 con 0x7efc601d3d40 2024-03-29T11:43:59.533 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.519+0000 7efc577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:59.533 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.519+0000 7efc577fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:59.533 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.519+0000 7efc577fe640 10 monclient(hunting): dump: 2024-03-29T11:43:59.533 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:43:59.533 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:59.533 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:59.533 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:59.533 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:59.533 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:59.533 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:43:59.533 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:43:59.534 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:43:59.534 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:59.534 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.519+0000 7efc577fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:59.534 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.520+0000 7efc577fe640 10 client.?.objecter ms_handle_connect 0x7efc6013f650 2024-03-29T11:43:59.534 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.520+0000 7efc577fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:59.534 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.520+0000 7efc577fe640 1 -- 172.21.15.72:0/102033489 <== mon.2 v1:172.21.15.72:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3322289390 0 0) 0x7efc601350f0 con 0x7efc601d3d40 2024-03-29T11:43:59.534 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.520+0000 7efc577fe640 1 -- 172.21.15.72:0/102033489 >> v1:172.21.15.89:6789/0 conn(0x7efc6013f650 legacy=0x7efc601d4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:59.534 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.520+0000 7efc577fe640 1 -- 172.21.15.72:0/102033489 >> v1:172.21.15.72:6789/0 conn(0x7efc600a9d70 legacy=0x7efc601d7a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:43:59.534 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.521+0000 7efc577fe640 10 monclient: _finish_hunting 0 2024-03-29T11:43:59.534 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.521+0000 7efc577fe640 1 monclient: found mon.c 2024-03-29T11:43:59.534 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.521+0000 7efc577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:59.534 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.521+0000 7efc577fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:43:59.534 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.521+0000 7efc577fe640 1 -- 172.21.15.72:0/102033489 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc601d7240 con 0x7efc601d3d40 2024-03-29T11:43:59.534 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.521+0000 7efc577fe640 10 monclient: _finish_auth 0 2024-03-29T11:43:59.535 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.521+0000 7efc577fe640 10 monclient: _check_auth_tickets 2024-03-29T11:43:59.535 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.521+0000 7efc577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:59.535 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.522+0000 7efc65523640 5 monclient: authenticate success, global_id 4559 2024-03-29T11:43:59.535 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.522+0000 7efc65523640 10 monclient: _renew_subs 2024-03-29T11:43:59.535 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.522+0000 7efc65523640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:43:59.535 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.522+0000 7efc65523640 1 -- 172.21.15.72:0/102033489 --> v1:172.21.15.72:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc601d7520 con 0x7efc601d3d40 2024-03-29T11:43:59.535 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.522+0000 7efc65523640 10 client.4559.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:59.535 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.522+0000 7efc65523640 10 monclient: _renew_subs 2024-03-29T11:43:59.535 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.522+0000 7efc65523640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:43:59.535 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.522+0000 7efc65523640 1 -- 172.21.15.72:0/102033489 --> v1:172.21.15.72:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efc601d4e50 con 0x7efc601d3d40 2024-03-29T11:43:59.535 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.522+0000 7efc65523640 1 librados: init done 2024-03-29T11:43:59.535 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.523+0000 7efc577fe640 1 -- 172.21.15.72:0/102033489 <== mon.2 v1:172.21.15.72:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc58003970 con 0x7efc601d3d40 2024-03-29T11:43:59.535 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.523+0000 7efc577fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:59.535 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.523+0000 7efc577fe640 1 -- 172.21.15.72:0/102033489 <== mon.2 v1:172.21.15.72:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7efc58003d80 con 0x7efc601d3d40 2024-03-29T11:43:59.535 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.523+0000 7efc577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:59.535 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.523+0000 7efc577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:59.536 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.523+0000 7efc577fe640 10 monclient: dump: 2024-03-29T11:43:59.536 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:43:59.536 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:43:59.536 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:59.536 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:43:59.536 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:59.536 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:43:59.536 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:43:59.536 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:43:59.536 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:43:59.536 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:43:59.536 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.523+0000 7efc5dffb640 4 set_mon_vals no callback set 2024-03-29T11:43:59.536 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.523+0000 7efc65523640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:59.536 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.525+0000 7efc577fe640 1 -- 172.21.15.72:0/102033489 <== mon.2 v1:172.21.15.72:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7efc58003ee0 con 0x7efc601d3d40 2024-03-29T11:43:59.537 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.525+0000 7efc65523640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:59.537 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.525+0000 7efc65523640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:43:59.537 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.525+0000 7efc65523640 1 -- 172.21.15.72:0/102033489 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc44005080 con 0x7efc601d3d40 2024-03-29T11:43:59.537 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.525+0000 7efc577fe640 1 -- 172.21.15.72:0/102033489 <== mon.2 v1:172.21.15.72:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7efc58084d90 con 0x7efc601d3d40 2024-03-29T11:43:59.537 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.525+0000 7efc577fe640 10 client.4559.objecter ms_dispatch 0x7efc60000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:59.537 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.525+0000 7efc577fe640 3 client.4559.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:59.537 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.525+0000 7efc577fe640 3 client.4559.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:59.537 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.526+0000 7efc577fe640 20 client.4559.objecter dump_active .. 0 homeless 2024-03-29T11:43:59.537 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.526+0000 7efc5effd640 10 monclient: get_auth_request con 0x7efc4006da80 auth_method 0 2024-03-29T11:43:59.537 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.526+0000 7efc577fe640 10 client.4559.objecter ms_handle_connect 0x7efc4006da80 2024-03-29T11:43:59.538 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.534+0000 7efc577fe640 1 -- 172.21.15.72:0/102033489 <== mon.2 v1:172.21.15.72:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efc58050670 con 0x7efc601d3d40 2024-03-29T11:43:59.538 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.534+0000 7efc577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:59.538 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.534+0000 7efc577fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:59.845 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.842+0000 7efc65523640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:59.846 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.842+0000 7efc65523640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:59.846 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.842+0000 7efc65523640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:43:59.846 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.842+0000 7efc65523640 1 -- 172.21.15.72:0/102033489 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc44005330 con 0x7efc601d3d40 2024-03-29T11:43:59.846 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.842+0000 7efc577fe640 1 -- 172.21.15.72:0/102033489 <== mon.2 v1:172.21.15.72:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7efc580525a0 con 0x7efc601d3d40 2024-03-29T11:43:59.846 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.843+0000 7efc577fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:59.846 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.843+0000 7efc577fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:59.847 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:43:59.847 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:59.850 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.845+0000 7efc65523640 10 librados: watch_flush enter 2024-03-29T11:43:59.850 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.846+0000 7efc65523640 10 librados: watch_flush exit 2024-03-29T11:43:59.850 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.846+0000 7efc65523640 20 client.4559.objecter shutdown clearing up homeless session... 2024-03-29T11:43:59.850 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.846+0000 7efc65523640 10 client.4559.objecter successfully canceled tick 2024-03-29T11:43:59.850 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.846+0000 7efc65523640 1 -- 172.21.15.72:0/102033489 >> v1:172.21.15.72:6816/1578866102 conn(0x7efc4006da80 legacy=0x7efc4006ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:59.850 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.846+0000 7efc65523640 10 monclient: shutdown 2024-03-29T11:43:59.850 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.846+0000 7efc65523640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:59.851 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.846+0000 7efc65523640 1 -- 172.21.15.72:0/102033489 >> v1:172.21.15.72:6790/0 conn(0x7efc601d3d40 legacy=0x7efc60133400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:59.851 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.846+0000 7efc65523640 1 -- 172.21.15.72:0/102033489 shutdown_connections 2024-03-29T11:43:59.851 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.846+0000 7efc65523640 1 -- 172.21.15.72:0/102033489 >> 172.21.15.72:0/102033489 conn(0x7efc60066a00 msgr2=0x7efc60066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:59.851 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.846+0000 7efc65523640 1 -- 172.21.15.72:0/102033489 shutdown_connections 2024-03-29T11:43:59.851 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.846+0000 7efc65523640 1 -- 172.21.15.72:0/102033489 wait complete. 2024-03-29T11:43:59.851 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:43:59.846+0000 7efc65523640 1 librados: shutdown 2024-03-29T11:43:59.866 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:06.673 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:06.821 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.816+0000 7ff42f79d640 10 monclient: get_monmap_and_config 2024-03-29T11:44:06.821 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.816+0000 7ff42f79d640 10 monclient: build_initial_monmap 2024-03-29T11:44:06.821 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.816+0000 7ff42f79d640 10 monclient: monmap: 2024-03-29T11:44:06.821 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:44:06.821 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:44:06.822 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:44:06.817558+0000 2024-03-29T11:44:06.822 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:44:06.817558+0000 2024-03-29T11:44:06.822 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:06.822 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:06.822 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:44:06.822 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:44:06.822 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:44:06.822 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:06.822 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.817+0000 7ff42f79d640 1 Processor -- start 2024-03-29T11:44:06.822 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.817+0000 7ff42f79d640 1 -- start start 2024-03-29T11:44:06.823 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.817+0000 7ff42f79d640 10 monclient: init 2024-03-29T11:44:06.825 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.818+0000 7ff42f79d640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:06.825 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.818+0000 7ff42f79d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:44:06.826 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.818+0000 7ff42f79d640 10 monclient(hunting): picked mon.noname-b con 0x7ff4280b0500 addr v1:172.21.15.72:6790/0 2024-03-29T11:44:06.826 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.818+0000 7ff42f79d640 10 monclient(hunting): picked mon.noname-a con 0x7ff4280b08e0 addr v1:172.21.15.72:6789/0 2024-03-29T11:44:06.826 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.818+0000 7ff42f79d640 10 monclient(hunting): picked mon.noname-c con 0x7ff4280aeca0 addr v1:172.21.15.89:6789/0 2024-03-29T11:44:06.826 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.818+0000 7ff42f79d640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff428140620 con 0x7ff4280b08e0 2024-03-29T11:44:06.826 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.818+0000 7ff42f79d640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff42813daf0 con 0x7ff4280aeca0 2024-03-29T11:44:06.826 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.818+0000 7ff42f79d640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4280ae180 con 0x7ff4280b0500 2024-03-29T11:44:06.826 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.818+0000 7ff42f79d640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:06.826 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.818+0000 7ff42f79d640 10 monclient(hunting): authenticate will time out at 1406.106715s 2024-03-29T11:44:06.826 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.819+0000 7ff42d539640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7ff4280b0500 0x7ff42813ff10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:60904/0 (socket says 172.21.15.72:60904) 2024-03-29T11:44:06.826 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.819+0000 7ff42d539640 1 -- 172.21.15.72:0/340673653 learned_addr learned my addr 172.21.15.72:0/340673653 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:44:06.826 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.820+0000 7ff41ffff640 1 -- 172.21.15.72:0/340673653 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1259381682 0 0) 0x7ff4280ae180 con 0x7ff4280b0500 2024-03-29T11:44:06.827 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.820+0000 7ff41ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:44:06.827 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.820+0000 7ff41ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:06.827 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.820+0000 7ff41ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:06.827 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.820+0000 7ff41ffff640 10 monclient(hunting): my global_id is 4562 2024-03-29T11:44:06.827 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.820+0000 7ff41ffff640 1 -- 172.21.15.72:0/340673653 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff414003520 con 0x7ff4280b0500 2024-03-29T11:44:06.827 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.821+0000 7ff41ffff640 1 -- 172.21.15.72:0/340673653 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 150468906 0 0) 0x7ff414003520 con 0x7ff4280b0500 2024-03-29T11:44:06.827 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.821+0000 7ff41ffff640 1 -- 172.21.15.72:0/340673653 >> v1:172.21.15.89:6789/0 conn(0x7ff4280aeca0 legacy=0x7ff42814b7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:44:06.827 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.821+0000 7ff41ffff640 1 -- 172.21.15.72:0/340673653 >> v1:172.21.15.72:6789/0 conn(0x7ff4280b08e0 legacy=0x7ff42813d3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:44:06.829 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.825+0000 7ff41ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:44:06.829 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.825+0000 7ff41ffff640 1 monclient: found mon.noname-b 2024-03-29T11:44:06.829 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.825+0000 7ff41ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:06.830 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.825+0000 7ff41ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.72:6790/0 2024-03-29T11:44:06.830 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.825+0000 7ff41ffff640 1 -- 172.21.15.72:0/340673653 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff428072510 con 0x7ff4280b0500 2024-03-29T11:44:06.830 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.825+0000 7ff41ffff640 10 monclient: _finish_auth 0 2024-03-29T11:44:06.831 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.825+0000 7ff41ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:44:06.831 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.825+0000 7ff41ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:06.831 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.825+0000 7ff42f79d640 5 monclient: authenticate success, global_id 4562 2024-03-29T11:44:06.832 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.825+0000 7ff42f79d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:06.832 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.825+0000 7ff41ffff640 1 -- 172.21.15.72:0/340673653 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7ff420001a10 con 0x7ff4280b0500 2024-03-29T11:44:06.832 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.825+0000 7ff41ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:06.832 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.825+0000 7ff41ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:06.832 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.825+0000 7ff41ffff640 10 monclient: dump: 2024-03-29T11:44:06.832 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:44:06.832 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:44:06.832 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:06.832 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:06.832 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:06.832 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:06.833 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:44:06.833 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:44:06.833 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:44:06.833 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:06.833 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.825+0000 7ff42f79d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:06.833 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.825+0000 7ff41ffff640 1 -- 172.21.15.72:0/340673653 <== mon.2 v1:172.21.15.72:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff420001ca0 con 0x7ff4280b0500 2024-03-29T11:44:06.833 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.825+0000 7ff41ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:06.833 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.825+0000 7ff42f79d640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:06.833 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.825+0000 7ff42f79d640 4 set_mon_vals no callback set 2024-03-29T11:44:06.833 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.826+0000 7ff41ffff640 1 -- 172.21.15.72:0/340673653 <== mon.2 v1:172.21.15.72:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7ff420002030 con 0x7ff4280b0500 2024-03-29T11:44:06.833 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.826+0000 7ff41ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:06.833 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.826+0000 7ff41ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:06.833 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.826+0000 7ff41ffff640 10 monclient: dump: 2024-03-29T11:44:06.833 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:44:06.833 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:44:06.833 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:06.834 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:06.834 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:06.834 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:06.834 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:44:06.834 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:44:06.834 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:44:06.834 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:06.834 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.826+0000 7ff42f79d640 10 monclient: shutdown 2024-03-29T11:44:06.834 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.826+0000 7ff42f79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:06.834 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.826+0000 7ff42f79d640 1 -- 172.21.15.72:0/340673653 >> v1:172.21.15.72:6790/0 conn(0x7ff4280b0500 legacy=0x7ff42813ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:06.834 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.826+0000 7ff42f79d640 1 -- 172.21.15.72:0/340673653 shutdown_connections 2024-03-29T11:44:06.834 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.826+0000 7ff42f79d640 1 -- 172.21.15.72:0/340673653 >> 172.21.15.72:0/340673653 conn(0x7ff42806b660 msgr2=0x7ff42806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:06.834 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.827+0000 7ff42f79d640 1 -- 172.21.15.72:0/340673653 shutdown_connections 2024-03-29T11:44:06.834 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.827+0000 7ff42f79d640 1 -- 172.21.15.72:0/340673653 wait complete. 2024-03-29T11:44:06.838 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.828+0000 7ff42f79d640 10 monclient: build_initial_monmap 2024-03-29T11:44:06.839 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.828+0000 7ff42f79d640 10 monclient: monmap: 2024-03-29T11:44:06.839 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:44:06.839 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:06.839 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:44:06.839 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:44:06.839 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:06.840 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:06.840 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:44:06.840 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:44:06.840 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:44:06.840 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:06.840 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.828+0000 7ff42f79d640 1 librados: starting msgr at 2024-03-29T11:44:06.840 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.828+0000 7ff42f79d640 1 librados: starting objecter 2024-03-29T11:44:06.840 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.828+0000 7ff42f79d640 1 Processor -- start 2024-03-29T11:44:06.840 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.828+0000 7ff42f79d640 1 -- start start 2024-03-29T11:44:06.840 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.828+0000 7ff42f79d640 1 librados: setting wanted keys 2024-03-29T11:44:06.840 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.828+0000 7ff42f79d640 1 librados: calling monclient init 2024-03-29T11:44:06.840 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.828+0000 7ff42f79d640 10 monclient: init 2024-03-29T11:44:06.840 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.829+0000 7ff42f79d640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:06.840 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.829+0000 7ff42f79d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:44:06.840 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.829+0000 7ff42f79d640 10 monclient(hunting): picked mon.noname-c con 0x7ff4280aeca0 addr v1:172.21.15.72:6790/0 2024-03-29T11:44:06.840 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.829+0000 7ff42f79d640 10 monclient(hunting): picked mon.noname-a con 0x7ff4280b0500 addr v1:172.21.15.72:6789/0 2024-03-29T11:44:06.841 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.829+0000 7ff42f79d640 10 monclient(hunting): picked mon.noname-b con 0x7ff4281dc120 addr v1:172.21.15.89:6789/0 2024-03-29T11:44:06.841 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.829+0000 7ff42f79d640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff42814f840 con 0x7ff4280b0500 2024-03-29T11:44:06.841 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.829+0000 7ff42f79d640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4281db4f0 con 0x7ff4281dc120 2024-03-29T11:44:06.841 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.829+0000 7ff42f79d640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff42813c6d0 con 0x7ff4280aeca0 2024-03-29T11:44:06.841 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.829+0000 7ff42f79d640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:06.841 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.829+0000 7ff42f79d640 10 monclient(hunting): authenticate will time out at 1406.117621s 2024-03-29T11:44:06.841 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.829+0000 7ff42d539640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7ff4280aeca0 0x7ff4281dba10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:36300/0 (socket says 172.21.15.72:36300) 2024-03-29T11:44:06.841 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.829+0000 7ff42d539640 1 -- 172.21.15.72:0/3240940729 learned_addr learned my addr 172.21.15.72:0/3240940729 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:44:06.841 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.831+0000 7ff41dffb640 10 client.?.objecter ms_handle_connect 0x7ff4280aeca0 2024-03-29T11:44:06.841 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.831+0000 7ff41dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:06.841 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.831+0000 7ff41dffb640 1 -- 172.21.15.72:0/3240940729 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2081115530 0 0) 0x7ff42813c6d0 con 0x7ff4280aeca0 2024-03-29T11:44:06.841 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.831+0000 7ff41dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:44:06.841 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.831+0000 7ff41dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:06.842 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.831+0000 7ff41dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:06.842 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.831+0000 7ff41dffb640 10 monclient(hunting): my global_id is 4565 2024-03-29T11:44:06.842 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.831+0000 7ff41dffb640 1 -- 172.21.15.72:0/3240940729 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff40c004eb0 con 0x7ff4280aeca0 2024-03-29T11:44:06.842 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.832+0000 7ff41dffb640 1 -- 172.21.15.72:0/3240940729 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2560699328 0 0) 0x7ff40c004eb0 con 0x7ff4280aeca0 2024-03-29T11:44:06.842 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.832+0000 7ff41dffb640 1 -- 172.21.15.72:0/3240940729 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff42813c6d0 con 0x7ff4280aeca0 2024-03-29T11:44:06.842 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.832+0000 7ff41dffb640 1 -- 172.21.15.72:0/3240940729 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7ff420001990 con 0x7ff4280aeca0 2024-03-29T11:44:06.842 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.832+0000 7ff41dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:06.842 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.832+0000 7ff41dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:44:06.842 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.832+0000 7ff41dffb640 10 monclient(hunting): dump: 2024-03-29T11:44:06.842 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:44:06.842 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:44:06.842 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:06.843 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:06.843 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:06.843 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:06.843 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:44:06.843 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:44:06.843 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:44:06.843 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:06.843 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.832+0000 7ff41dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:06.843 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.833+0000 7ff41dffb640 1 -- 172.21.15.72:0/3240940729 <== mon.2 v1:172.21.15.72:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 336438078 0 0) 0x7ff42813c6d0 con 0x7ff4280aeca0 2024-03-29T11:44:06.843 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.833+0000 7ff41dffb640 1 -- 172.21.15.72:0/3240940729 >> v1:172.21.15.89:6789/0 conn(0x7ff4281dc120 legacy=0x7ff42813bcf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:44:06.843 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.834+0000 7ff41dffb640 1 -- 172.21.15.72:0/3240940729 >> v1:172.21.15.72:6789/0 conn(0x7ff4280b0500 legacy=0x7ff42813b5e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:44:06.845 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.834+0000 7ff41dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:44:06.845 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.834+0000 7ff41dffb640 1 monclient: found mon.c 2024-03-29T11:44:06.845 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.834+0000 7ff41dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:06.845 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.834+0000 7ff41dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:44:06.845 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.834+0000 7ff41dffb640 1 -- 172.21.15.72:0/3240940729 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff42813c860 con 0x7ff4280aeca0 2024-03-29T11:44:06.845 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.835+0000 7ff41dffb640 10 monclient: _finish_auth 0 2024-03-29T11:44:06.845 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.835+0000 7ff41dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:44:06.846 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.835+0000 7ff41dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:06.846 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.835+0000 7ff42f79d640 5 monclient: authenticate success, global_id 4565 2024-03-29T11:44:06.846 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.835+0000 7ff42f79d640 10 monclient: _renew_subs 2024-03-29T11:44:06.846 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.835+0000 7ff42f79d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:44:06.846 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.835+0000 7ff42f79d640 1 -- 172.21.15.72:0/3240940729 --> v1:172.21.15.72:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff42814e900 con 0x7ff4280aeca0 2024-03-29T11:44:06.846 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.836+0000 7ff42f79d640 10 client.4565.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:06.846 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.836+0000 7ff42f79d640 10 monclient: _renew_subs 2024-03-29T11:44:06.846 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.836+0000 7ff42f79d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:44:06.846 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.836+0000 7ff42f79d640 1 -- 172.21.15.72:0/3240940729 --> v1:172.21.15.72:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff428070db0 con 0x7ff4280aeca0 2024-03-29T11:44:06.846 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.836+0000 7ff42f79d640 1 librados: init done 2024-03-29T11:44:06.846 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.836+0000 7ff41dffb640 1 -- 172.21.15.72:0/3240940729 <== mon.2 v1:172.21.15.72:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff420004f70 con 0x7ff4280aeca0 2024-03-29T11:44:06.847 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.836+0000 7ff41dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:06.847 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.836+0000 7ff41f7fe640 4 set_mon_vals no callback set 2024-03-29T11:44:06.847 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.836+0000 7ff41dffb640 1 -- 172.21.15.72:0/3240940729 <== mon.2 v1:172.21.15.72:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7ff4200052c0 con 0x7ff4280aeca0 2024-03-29T11:44:06.847 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.836+0000 7ff41dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:06.847 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.836+0000 7ff41dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:06.847 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.836+0000 7ff41dffb640 10 monclient: dump: 2024-03-29T11:44:06.847 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:44:06.847 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:44:06.847 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:06.847 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:06.847 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:06.847 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:06.847 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:44:06.848 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:44:06.848 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:44:06.848 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:06.848 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.837+0000 7ff42f79d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:06.848 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.837+0000 7ff42f79d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:06.848 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.837+0000 7ff42f79d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:44:06.848 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.837+0000 7ff42f79d640 1 -- 172.21.15.72:0/3240940729 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff408005080 con 0x7ff4280aeca0 2024-03-29T11:44:06.848 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.838+0000 7ff41dffb640 1 -- 172.21.15.72:0/3240940729 <== mon.2 v1:172.21.15.72:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7ff42001b290 con 0x7ff4280aeca0 2024-03-29T11:44:06.848 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.841+0000 7ff41dffb640 1 -- 172.21.15.72:0/3240940729 <== mon.2 v1:172.21.15.72:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7ff420085050 con 0x7ff4280aeca0 2024-03-29T11:44:06.848 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.841+0000 7ff41dffb640 10 client.4565.objecter ms_dispatch 0x7ff428000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:06.848 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.841+0000 7ff42cd38640 10 monclient: get_auth_request con 0x7ff40c06dbe0 auth_method 0 2024-03-29T11:44:06.848 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.841+0000 7ff41dffb640 3 client.4565.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:06.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.841+0000 7ff41dffb640 3 client.4565.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:06.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.841+0000 7ff41dffb640 20 client.4565.objecter dump_active .. 0 homeless 2024-03-29T11:44:06.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.841+0000 7ff41dffb640 10 client.4565.objecter ms_handle_connect 0x7ff40c06dbe0 2024-03-29T11:44:06.851 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.847+0000 7ff41dffb640 1 -- 172.21.15.72:0/3240940729 <== mon.2 v1:172.21.15.72:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff420050930 con 0x7ff4280aeca0 2024-03-29T11:44:06.851 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.847+0000 7ff41dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:06.851 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:06.847+0000 7ff41dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:07.183 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:07.179+0000 7ff42f79d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:07.183 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:07.179+0000 7ff42f79d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:07.183 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:07.179+0000 7ff42f79d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:44:07.183 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:07.179+0000 7ff42f79d640 1 -- 172.21.15.72:0/3240940729 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff408005330 con 0x7ff4280aeca0 2024-03-29T11:44:07.185 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:07.182+0000 7ff41dffb640 1 -- 172.21.15.72:0/3240940729 <== mon.2 v1:172.21.15.72:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7ff420052860 con 0x7ff4280aeca0 2024-03-29T11:44:07.185 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:07.182+0000 7ff41dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:07.186 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:07.182+0000 7ff41dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:07.186 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:44:07.186 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:07.189 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:07.185+0000 7ff42f79d640 10 librados: watch_flush enter 2024-03-29T11:44:07.189 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:07.185+0000 7ff42f79d640 10 librados: watch_flush exit 2024-03-29T11:44:07.189 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:07.185+0000 7ff42f79d640 20 client.4565.objecter shutdown clearing up homeless session... 2024-03-29T11:44:07.190 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:07.185+0000 7ff42f79d640 10 client.4565.objecter successfully canceled tick 2024-03-29T11:44:07.190 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:07.185+0000 7ff42f79d640 1 -- 172.21.15.72:0/3240940729 >> v1:172.21.15.72:6816/1578866102 conn(0x7ff40c06dbe0 legacy=0x7ff40c0700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:07.190 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:07.185+0000 7ff42f79d640 10 monclient: shutdown 2024-03-29T11:44:07.190 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:07.185+0000 7ff42f79d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:07.190 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:07.185+0000 7ff42f79d640 1 -- 172.21.15.72:0/3240940729 >> v1:172.21.15.72:6790/0 conn(0x7ff4280aeca0 legacy=0x7ff4281dba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:07.190 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:07.185+0000 7ff42f79d640 1 -- 172.21.15.72:0/3240940729 shutdown_connections 2024-03-29T11:44:07.190 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:07.185+0000 7ff42f79d640 1 -- 172.21.15.72:0/3240940729 >> 172.21.15.72:0/3240940729 conn(0x7ff42806b660 msgr2=0x7ff4280b11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:07.190 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:07.185+0000 7ff42f79d640 1 -- 172.21.15.72:0/3240940729 shutdown_connections 2024-03-29T11:44:07.190 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:07.185+0000 7ff42f79d640 1 -- 172.21.15.72:0/3240940729 wait complete. 2024-03-29T11:44:07.190 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:07.186+0000 7ff42f79d640 1 librados: shutdown 2024-03-29T11:44:07.209 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:14.017 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:14.163 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.158+0000 7f28425b5640 10 monclient: get_monmap_and_config 2024-03-29T11:44:14.163 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.158+0000 7f28425b5640 10 monclient: build_initial_monmap 2024-03-29T11:44:14.163 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.158+0000 7f28425b5640 10 monclient: monmap: 2024-03-29T11:44:14.163 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:44:14.163 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:44:14.163 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:44:14.159186+0000 2024-03-29T11:44:14.163 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:44:14.159186+0000 2024-03-29T11:44:14.163 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:14.163 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:14.163 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:44:14.163 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:44:14.164 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:44:14.164 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:14.164 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.159+0000 7f28425b5640 1 Processor -- start 2024-03-29T11:44:14.164 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.159+0000 7f28425b5640 1 -- start start 2024-03-29T11:44:14.164 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.159+0000 7f28425b5640 10 monclient: init 2024-03-29T11:44:14.164 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.160+0000 7f28425b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:14.164 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.160+0000 7f28425b5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:44:14.167 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.162+0000 7f28425b5640 10 monclient(hunting): picked mon.noname-a con 0x7f283c0b0860 addr v1:172.21.15.72:6789/0 2024-03-29T11:44:14.167 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.162+0000 7f28425b5640 10 monclient(hunting): picked mon.noname-b con 0x7f283c13ef30 addr v1:172.21.15.72:6790/0 2024-03-29T11:44:14.167 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.162+0000 7f28425b5640 10 monclient(hunting): picked mon.noname-c con 0x7f283c06d9d0 addr v1:172.21.15.89:6789/0 2024-03-29T11:44:14.168 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.162+0000 7f28425b5640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f283c0b0c40 con 0x7f283c0b0860 2024-03-29T11:44:14.168 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.162+0000 7f28425b5640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f283c0aed10 con 0x7f283c06d9d0 2024-03-29T11:44:14.168 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.162+0000 7f28425b5640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f283c0ab1b0 con 0x7f283c13ef30 2024-03-29T11:44:14.168 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.162+0000 7f28425b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:14.168 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.162+0000 7f28425b5640 10 monclient(hunting): authenticate will time out at 1413.450964s 2024-03-29T11:44:14.168 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.163+0000 7f283b7fe640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f283c13ef30 0x7f283c0af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:57282/0 (socket says 172.21.15.72:57282) 2024-03-29T11:44:14.168 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.163+0000 7f283b7fe640 1 -- 172.21.15.72:0/3475598415 learned_addr learned my addr 172.21.15.72:0/3475598415 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:44:14.168 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.163+0000 7f283affd640 1 -- 172.21.15.72:0/3475598415 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 826334809 0 0) 0x7f283c0ab1b0 con 0x7f283c13ef30 2024-03-29T11:44:14.168 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.163+0000 7f283affd640 10 monclient(hunting): _negotiate 2024-03-29T11:44:14.168 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.163+0000 7f283affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:14.168 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.163+0000 7f283affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:14.168 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.163+0000 7f283affd640 10 monclient(hunting): my global_id is 4568 2024-03-29T11:44:14.168 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.163+0000 7f283affd640 1 -- 172.21.15.72:0/3475598415 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2824003520 con 0x7f283c13ef30 2024-03-29T11:44:14.168 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.164+0000 7f283affd640 1 -- 172.21.15.72:0/3475598415 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3588672511 0 0) 0x7f2824003520 con 0x7f283c13ef30 2024-03-29T11:44:14.168 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.164+0000 7f283affd640 1 -- 172.21.15.72:0/3475598415 >> v1:172.21.15.89:6789/0 conn(0x7f283c06d9d0 legacy=0x7f283c14a300 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:44:14.169 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.165+0000 7f283affd640 1 -- 172.21.15.72:0/3475598415 >> v1:172.21.15.72:6789/0 conn(0x7f283c0b0860 legacy=0x7f283c13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:44:14.169 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.165+0000 7f283affd640 10 monclient: _finish_hunting 0 2024-03-29T11:44:14.169 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.165+0000 7f283affd640 1 monclient: found mon.noname-b 2024-03-29T11:44:14.169 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.165+0000 7f283affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:14.169 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.165+0000 7f283affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.72:6790/0 2024-03-29T11:44:14.169 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.165+0000 7f283affd640 1 -- 172.21.15.72:0/3475598415 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f283c0ab490 con 0x7f283c13ef30 2024-03-29T11:44:14.170 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.165+0000 7f283affd640 10 monclient: _finish_auth 0 2024-03-29T11:44:14.170 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.165+0000 7f283affd640 10 monclient: _check_auth_tickets 2024-03-29T11:44:14.170 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.165+0000 7f283affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:14.170 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.165+0000 7f283affd640 1 -- 172.21.15.72:0/3475598415 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f2830001690 con 0x7f283c13ef30 2024-03-29T11:44:14.170 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.165+0000 7f283affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:14.170 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.165+0000 7f283affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:14.170 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.165+0000 7f283affd640 10 monclient: dump: 2024-03-29T11:44:14.170 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:44:14.170 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:44:14.170 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:14.170 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:14.170 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:14.170 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:14.170 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:44:14.170 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:44:14.170 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:44:14.171 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:14.171 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.165+0000 7f28425b5640 5 monclient: authenticate success, global_id 4568 2024-03-29T11:44:14.171 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.165+0000 7f28425b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:14.171 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.168+0000 7f283affd640 1 -- 172.21.15.72:0/3475598415 <== mon.2 v1:172.21.15.72:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f28300018f0 con 0x7f283c13ef30 2024-03-29T11:44:14.172 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.168+0000 7f283affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:14.172 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.168+0000 7f283affd640 1 -- 172.21.15.72:0/3475598415 <== mon.2 v1:172.21.15.72:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f2830001d40 con 0x7f283c13ef30 2024-03-29T11:44:14.172 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.168+0000 7f283affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:14.172 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.168+0000 7f283affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:14.172 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.168+0000 7f283affd640 10 monclient: dump: 2024-03-29T11:44:14.172 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:44:14.172 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:44:14.172 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:14.172 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:14.172 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:14.172 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:14.172 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:44:14.172 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:44:14.172 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:44:14.172 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:14.173 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.168+0000 7f28425b5640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:14.173 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.168+0000 7f28425b5640 4 set_mon_vals no callback set 2024-03-29T11:44:14.173 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.169+0000 7f28425b5640 10 monclient: shutdown 2024-03-29T11:44:14.173 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.169+0000 7f28425b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:14.173 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.169+0000 7f28425b5640 1 -- 172.21.15.72:0/3475598415 >> v1:172.21.15.72:6790/0 conn(0x7f283c13ef30 legacy=0x7f283c0af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:14.175 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.172+0000 7f28425b5640 1 -- 172.21.15.72:0/3475598415 shutdown_connections 2024-03-29T11:44:14.176 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.172+0000 7f28425b5640 1 -- 172.21.15.72:0/3475598415 >> 172.21.15.72:0/3475598415 conn(0x7f283c06b660 msgr2=0x7f283c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:14.180 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.175+0000 7f28425b5640 1 -- 172.21.15.72:0/3475598415 shutdown_connections 2024-03-29T11:44:14.180 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.175+0000 7f28425b5640 1 -- 172.21.15.72:0/3475598415 wait complete. 2024-03-29T11:44:14.180 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.176+0000 7f28425b5640 10 monclient: build_initial_monmap 2024-03-29T11:44:14.180 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.176+0000 7f28425b5640 10 monclient: monmap: 2024-03-29T11:44:14.180 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:44:14.180 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:14.180 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:44:14.180 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:44:14.180 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:14.181 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:14.181 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:44:14.181 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:44:14.181 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:44:14.181 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:14.181 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.176+0000 7f28425b5640 1 librados: starting msgr at 2024-03-29T11:44:14.182 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.177+0000 7f28425b5640 1 librados: starting objecter 2024-03-29T11:44:14.182 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.177+0000 7f28425b5640 1 Processor -- start 2024-03-29T11:44:14.182 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.177+0000 7f28425b5640 1 -- start start 2024-03-29T11:44:14.182 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.177+0000 7f28425b5640 1 librados: setting wanted keys 2024-03-29T11:44:14.182 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.177+0000 7f28425b5640 1 librados: calling monclient init 2024-03-29T11:44:14.182 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.177+0000 7f28425b5640 10 monclient: init 2024-03-29T11:44:14.182 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.178+0000 7f28425b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:14.182 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.178+0000 7f28425b5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:44:14.182 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.178+0000 7f28425b5640 10 monclient(hunting): picked mon.noname-c con 0x7f283c06d9d0 addr v1:172.21.15.72:6790/0 2024-03-29T11:44:14.185 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.179+0000 7f28425b5640 10 monclient(hunting): picked mon.noname-a con 0x7f283c0b0860 addr v1:172.21.15.72:6789/0 2024-03-29T11:44:14.185 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.179+0000 7f28425b5640 10 monclient(hunting): picked mon.noname-b con 0x7f283c13ef30 addr v1:172.21.15.89:6789/0 2024-03-29T11:44:14.185 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.179+0000 7f28425b5640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f283c071cb0 con 0x7f283c0b0860 2024-03-29T11:44:14.185 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.179+0000 7f28425b5640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f283c0708a0 con 0x7f283c13ef30 2024-03-29T11:44:14.185 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.179+0000 7f28425b5640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f283c06f710 con 0x7f283c06d9d0 2024-03-29T11:44:14.185 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.179+0000 7f28425b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:14.185 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.179+0000 7f28425b5640 10 monclient(hunting): authenticate will time out at 1413.467084s 2024-03-29T11:44:14.185 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.180+0000 7f283bfff640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f283c06d9d0 0x7f283c071030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:60422/0 (socket says 172.21.15.72:60422) 2024-03-29T11:44:14.185 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.180+0000 7f283bfff640 1 -- 172.21.15.72:0/1927118653 learned_addr learned my addr 172.21.15.72:0/1927118653 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:44:14.185 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.180+0000 7f2838ff9640 10 client.?.objecter ms_handle_connect 0x7f283c06d9d0 2024-03-29T11:44:14.185 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.180+0000 7f2838ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:14.185 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.180+0000 7f2838ff9640 1 -- 172.21.15.72:0/1927118653 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 732606793 0 0) 0x7f283c06f710 con 0x7f283c06d9d0 2024-03-29T11:44:14.186 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.181+0000 7f2838ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:44:14.186 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.181+0000 7f2838ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:14.186 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.181+0000 7f2838ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:14.186 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.181+0000 7f2838ff9640 10 monclient(hunting): my global_id is 4571 2024-03-29T11:44:14.186 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.181+0000 7f2838ff9640 1 -- 172.21.15.72:0/1927118653 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2828004eb0 con 0x7f283c06d9d0 2024-03-29T11:44:14.190 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.182+0000 7f2838ff9640 1 -- 172.21.15.72:0/1927118653 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3718834164 0 0) 0x7f2828004eb0 con 0x7f283c06d9d0 2024-03-29T11:44:14.190 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.182+0000 7f2838ff9640 1 -- 172.21.15.72:0/1927118653 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f283c06f710 con 0x7f283c06d9d0 2024-03-29T11:44:14.190 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.182+0000 7f2838ff9640 10 client.?.objecter ms_handle_connect 0x7f283c0b0860 2024-03-29T11:44:14.190 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.182+0000 7f2838ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:14.190 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.182+0000 7f2838ff9640 1 -- 172.21.15.72:0/1927118653 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f282c001960 con 0x7f283c06d9d0 2024-03-29T11:44:14.190 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.182+0000 7f2838ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:14.190 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.182+0000 7f2838ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:44:14.191 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.182+0000 7f2838ff9640 10 monclient(hunting): dump: 2024-03-29T11:44:14.191 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:44:14.191 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:44:14.191 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:14.191 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:14.191 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:14.191 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:14.191 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:44:14.191 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:44:14.191 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:44:14.191 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:14.191 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.183+0000 7f2838ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:14.191 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.184+0000 7f2838ff9640 10 client.?.objecter ms_handle_connect 0x7f283c13ef30 2024-03-29T11:44:14.191 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.184+0000 7f2838ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:14.192 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.184+0000 7f2838ff9640 1 -- 172.21.15.72:0/1927118653 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3938076656 0 0) 0x7f283c071cb0 con 0x7f283c0b0860 2024-03-29T11:44:14.192 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.184+0000 7f2838ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:44:14.192 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.184+0000 7f2838ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:14.192 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.184+0000 7f2838ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:14.192 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.184+0000 7f2838ff9640 10 monclient(hunting): my global_id is 4545 2024-03-29T11:44:14.192 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.184+0000 7f2838ff9640 1 -- 172.21.15.72:0/1927118653 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2828004eb0 con 0x7f283c0b0860 2024-03-29T11:44:14.192 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.184+0000 7f2838ff9640 1 -- 172.21.15.72:0/1927118653 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1130561684 0 0) 0x7f283c0708a0 con 0x7f283c13ef30 2024-03-29T11:44:14.192 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.184+0000 7f2838ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:44:14.192 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.184+0000 7f2838ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:14.192 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.184+0000 7f2838ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:14.192 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.184+0000 7f2838ff9640 10 monclient(hunting): my global_id is 4501 2024-03-29T11:44:14.192 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.184+0000 7f2838ff9640 1 -- 172.21.15.72:0/1927118653 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f283c071cb0 con 0x7f283c13ef30 2024-03-29T11:44:14.193 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.184+0000 7f2838ff9640 1 -- 172.21.15.72:0/1927118653 <== mon.2 v1:172.21.15.72:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 583071774 0 0) 0x7f283c06f710 con 0x7f283c06d9d0 2024-03-29T11:44:14.193 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.185+0000 7f2838ff9640 1 -- 172.21.15.72:0/1927118653 >> v1:172.21.15.89:6789/0 conn(0x7f283c13ef30 legacy=0x7f283c137250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:14.193 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.185+0000 7f2838ff9640 1 -- 172.21.15.72:0/1927118653 >> v1:172.21.15.72:6789/0 conn(0x7f283c0b0860 legacy=0x7f283c136b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:14.193 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.185+0000 7f2838ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:44:14.193 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.185+0000 7f2838ff9640 1 monclient: found mon.c 2024-03-29T11:44:14.193 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.185+0000 7f2838ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:14.193 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.185+0000 7f2838ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:44:14.193 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.185+0000 7f2838ff9640 1 -- 172.21.15.72:0/1927118653 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f283c06f8a0 con 0x7f283c06d9d0 2024-03-29T11:44:14.193 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.185+0000 7f2838ff9640 10 monclient: _finish_auth 0 2024-03-29T11:44:14.193 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.185+0000 7f2838ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:44:14.193 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.185+0000 7f2838ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:14.193 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.185+0000 7f28425b5640 5 monclient: authenticate success, global_id 4571 2024-03-29T11:44:14.193 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.185+0000 7f28425b5640 10 monclient: _renew_subs 2024-03-29T11:44:14.193 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.185+0000 7f28425b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:44:14.193 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.185+0000 7f28425b5640 1 -- 172.21.15.72:0/1927118653 --> v1:172.21.15.72:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f283c071740 con 0x7f283c06d9d0 2024-03-29T11:44:14.194 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.186+0000 7f28425b5640 10 client.4571.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:14.194 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.186+0000 7f28425b5640 10 monclient: _renew_subs 2024-03-29T11:44:14.194 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.186+0000 7f28425b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:44:14.194 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.186+0000 7f28425b5640 1 -- 172.21.15.72:0/1927118653 --> v1:172.21.15.72:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f283c0719f0 con 0x7f283c06d9d0 2024-03-29T11:44:14.194 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.186+0000 7f28425b5640 1 librados: init done 2024-03-29T11:44:14.196 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.187+0000 7f2838ff9640 1 -- 172.21.15.72:0/1927118653 <== mon.2 v1:172.21.15.72:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f282c001ac0 con 0x7f283c06d9d0 2024-03-29T11:44:14.197 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.187+0000 7f28425b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:14.197 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.187+0000 7f28425b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:14.197 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.187+0000 7f28425b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:44:14.197 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.187+0000 7f28425b5640 1 -- 172.21.15.72:0/1927118653 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f283c075f10 con 0x7f283c06d9d0 2024-03-29T11:44:14.197 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.187+0000 7f2838ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:14.197 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.187+0000 7f2838ff9640 1 -- 172.21.15.72:0/1927118653 <== mon.2 v1:172.21.15.72:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f282c0038a0 con 0x7f283c06d9d0 2024-03-29T11:44:14.197 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.187+0000 7f2838ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:14.197 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.187+0000 7f2838ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:14.197 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.187+0000 7f2838ff9640 10 monclient: dump: 2024-03-29T11:44:14.197 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:44:14.197 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:44:14.197 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:14.198 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:14.198 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:14.198 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:14.198 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:44:14.198 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:44:14.198 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:44:14.198 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:14.198 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.188+0000 7f283a7fc640 4 set_mon_vals no callback set 2024-03-29T11:44:14.198 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.189+0000 7f2838ff9640 1 -- 172.21.15.72:0/1927118653 <== mon.2 v1:172.21.15.72:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f282c003a00 con 0x7f283c06d9d0 2024-03-29T11:44:14.198 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.190+0000 7f2838ff9640 1 -- 172.21.15.72:0/1927118653 <== mon.2 v1:172.21.15.72:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f282c084890 con 0x7f283c06d9d0 2024-03-29T11:44:14.198 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.190+0000 7f2838ff9640 10 client.4571.objecter ms_dispatch 0x7f283c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:14.198 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.190+0000 7f2838ff9640 3 client.4571.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:14.198 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.190+0000 7f2838ff9640 3 client.4571.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:14.198 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.190+0000 7f2838ff9640 20 client.4571.objecter dump_active .. 0 homeless 2024-03-29T11:44:14.199 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.191+0000 7f283b7fe640 10 monclient: get_auth_request con 0x7f282806de30 auth_method 0 2024-03-29T11:44:14.199 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.191+0000 7f2838ff9640 10 client.4571.objecter ms_handle_connect 0x7f282806de30 2024-03-29T11:44:14.202 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.197+0000 7f2838ff9640 1 -- 172.21.15.72:0/1927118653 <== mon.2 v1:172.21.15.72:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f282c050170 con 0x7f283c06d9d0 2024-03-29T11:44:14.203 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.197+0000 7f2838ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:14.203 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.197+0000 7f2838ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:14.527 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.523+0000 7f28425b5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:14.527 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.523+0000 7f28425b5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:14.527 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.523+0000 7f28425b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:44:14.527 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.523+0000 7f28425b5640 1 -- 172.21.15.72:0/1927118653 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f283c0758a0 con 0x7f283c06d9d0 2024-03-29T11:44:14.528 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.524+0000 7f2838ff9640 1 -- 172.21.15.72:0/1927118653 <== mon.2 v1:172.21.15.72:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f283c0758a0 con 0x7f283c06d9d0 2024-03-29T11:44:14.528 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.524+0000 7f2838ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:14.528 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.524+0000 7f2838ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:14.530 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:44:14.530 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:14.533 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.529+0000 7f28425b5640 10 librados: watch_flush enter 2024-03-29T11:44:14.533 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.529+0000 7f28425b5640 10 librados: watch_flush exit 2024-03-29T11:44:14.533 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.529+0000 7f28425b5640 20 client.4571.objecter shutdown clearing up homeless session... 2024-03-29T11:44:14.533 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.529+0000 7f28425b5640 10 client.4571.objecter successfully canceled tick 2024-03-29T11:44:14.533 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.529+0000 7f28425b5640 1 -- 172.21.15.72:0/1927118653 >> v1:172.21.15.72:6816/1578866102 conn(0x7f282806de30 legacy=0x7f28280702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:14.534 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.529+0000 7f28425b5640 10 monclient: shutdown 2024-03-29T11:44:14.534 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.529+0000 7f28425b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:14.534 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.529+0000 7f28425b5640 1 -- 172.21.15.72:0/1927118653 >> v1:172.21.15.72:6790/0 conn(0x7f283c06d9d0 legacy=0x7f283c071030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:14.535 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.530+0000 7f28425b5640 1 -- 172.21.15.72:0/1927118653 shutdown_connections 2024-03-29T11:44:14.535 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.530+0000 7f28425b5640 1 -- 172.21.15.72:0/1927118653 >> 172.21.15.72:0/1927118653 conn(0x7f283c06b660 msgr2=0x7f283c13b5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:14.535 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.530+0000 7f28425b5640 1 -- 172.21.15.72:0/1927118653 shutdown_connections 2024-03-29T11:44:14.535 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.530+0000 7f28425b5640 1 -- 172.21.15.72:0/1927118653 wait complete. 2024-03-29T11:44:14.535 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:14.530+0000 7f28425b5640 1 librados: shutdown 2024-03-29T11:44:14.551 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:21.359 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:21.502 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.496+0000 7f25c2eff640 10 monclient: get_monmap_and_config 2024-03-29T11:44:21.509 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.496+0000 7f25c2eff640 10 monclient: build_initial_monmap 2024-03-29T11:44:21.509 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.496+0000 7f25c2eff640 10 monclient: monmap: 2024-03-29T11:44:21.509 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:44:21.509 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:44:21.509 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:44:21.498508+0000 2024-03-29T11:44:21.509 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:44:21.498508+0000 2024-03-29T11:44:21.509 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:21.509 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:21.509 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:44:21.509 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:44:21.510 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:44:21.510 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:21.510 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.497+0000 7f25c2eff640 1 Processor -- start 2024-03-29T11:44:21.510 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.497+0000 7f25c2eff640 1 -- start start 2024-03-29T11:44:21.510 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.497+0000 7f25c2eff640 10 monclient: init 2024-03-29T11:44:21.510 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.498+0000 7f25c2eff640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:21.510 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.498+0000 7f25c2eff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:44:21.510 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.498+0000 7f25c2eff640 10 monclient(hunting): picked mon.noname-a con 0x7f25bc0b4890 addr v1:172.21.15.72:6789/0 2024-03-29T11:44:21.510 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.498+0000 7f25c2eff640 10 monclient(hunting): picked mon.noname-b con 0x7f25bc152120 addr v1:172.21.15.72:6790/0 2024-03-29T11:44:21.510 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.498+0000 7f25c2eff640 10 monclient(hunting): picked mon.noname-c con 0x7f25bc071a60 addr v1:172.21.15.89:6789/0 2024-03-29T11:44:21.510 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.498+0000 7f25c2eff640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25bc06e7c0 con 0x7f25bc0b4890 2024-03-29T11:44:21.510 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.498+0000 7f25c2eff640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25bc06fa60 con 0x7f25bc071a60 2024-03-29T11:44:21.510 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.498+0000 7f25c2eff640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25bc06e2c0 con 0x7f25bc152120 2024-03-29T11:44:21.510 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.498+0000 7f25c2eff640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:21.511 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.498+0000 7f25c2eff640 10 monclient(hunting): authenticate will time out at 1420.787591s 2024-03-29T11:44:21.511 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.500+0000 7f25bbfff640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f25bc152120 0x7f25bc0b3b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:46596/0 (socket says 172.21.15.72:46596) 2024-03-29T11:44:21.511 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.500+0000 7f25bbfff640 1 -- 172.21.15.72:0/2274628383 learned_addr learned my addr 172.21.15.72:0/2274628383 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:44:21.511 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.500+0000 7f25bb7fe640 1 -- 172.21.15.72:0/2274628383 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 641328114 0 0) 0x7f25bc06e2c0 con 0x7f25bc152120 2024-03-29T11:44:21.511 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.500+0000 7f25bb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:44:21.511 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.500+0000 7f25bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:21.511 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.500+0000 7f25bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:21.511 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.500+0000 7f25bb7fe640 10 monclient(hunting): my global_id is 4574 2024-03-29T11:44:21.511 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.500+0000 7f25bb7fe640 1 -- 172.21.15.72:0/2274628383 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25a4003520 con 0x7f25bc152120 2024-03-29T11:44:21.511 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.501+0000 7f25bb7fe640 1 -- 172.21.15.72:0/2274628383 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2640986358 0 0) 0x7f25a4003520 con 0x7f25bc152120 2024-03-29T11:44:21.511 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.501+0000 7f25bb7fe640 1 -- 172.21.15.72:0/2274628383 >> v1:172.21.15.89:6789/0 conn(0x7f25bc071a60 legacy=0x7f25bc15a4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:44:21.511 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.503+0000 7f25bb7fe640 1 -- 172.21.15.72:0/2274628383 >> v1:172.21.15.72:6789/0 conn(0x7f25bc0b4890 legacy=0x7f25bc071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:21.511 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.503+0000 7f25bb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:44:21.511 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.503+0000 7f25bb7fe640 1 monclient: found mon.noname-b 2024-03-29T11:44:21.511 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.503+0000 7f25bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:21.512 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.503+0000 7f25bb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.72:6790/0 2024-03-29T11:44:21.512 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.503+0000 7f25bb7fe640 1 -- 172.21.15.72:0/2274628383 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25bc06e450 con 0x7f25bc152120 2024-03-29T11:44:21.512 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.503+0000 7f25bb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:44:21.512 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.503+0000 7f25bb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:44:21.512 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.503+0000 7f25bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:21.512 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.503+0000 7f25c2eff640 5 monclient: authenticate success, global_id 4574 2024-03-29T11:44:21.512 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.503+0000 7f25c2eff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:21.512 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.503+0000 7f25bb7fe640 1 -- 172.21.15.72:0/2274628383 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f25b0001690 con 0x7f25bc152120 2024-03-29T11:44:21.512 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.503+0000 7f25bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:21.512 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.503+0000 7f25bb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:21.512 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.503+0000 7f25bb7fe640 10 monclient: dump: 2024-03-29T11:44:21.512 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:44:21.512 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:44:21.512 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:21.512 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:21.513 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:21.513 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:21.513 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:44:21.513 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:44:21.513 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:44:21.513 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:21.513 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.503+0000 7f25c2eff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:21.513 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.503+0000 7f25bb7fe640 1 -- 172.21.15.72:0/2274628383 <== mon.2 v1:172.21.15.72:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25b00017f0 con 0x7f25bc152120 2024-03-29T11:44:21.513 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.503+0000 7f25bb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:21.513 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.503+0000 7f25c2eff640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:21.513 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.503+0000 7f25c2eff640 4 set_mon_vals no callback set 2024-03-29T11:44:21.514 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.504+0000 7f25c2eff640 10 monclient: shutdown 2024-03-29T11:44:21.514 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.504+0000 7f25c2eff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:21.514 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.504+0000 7f25c2eff640 1 -- 172.21.15.72:0/2274628383 >> v1:172.21.15.72:6790/0 conn(0x7f25bc152120 legacy=0x7f25bc0b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:21.514 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.504+0000 7f25c2eff640 1 -- 172.21.15.72:0/2274628383 shutdown_connections 2024-03-29T11:44:21.514 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.504+0000 7f25c2eff640 1 -- 172.21.15.72:0/2274628383 >> 172.21.15.72:0/2274628383 conn(0x7f25bc066a00 msgr2=0x7f25bc066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:21.514 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.504+0000 7f25c2eff640 1 -- 172.21.15.72:0/2274628383 shutdown_connections 2024-03-29T11:44:21.516 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.504+0000 7f25c2eff640 1 -- 172.21.15.72:0/2274628383 wait complete. 2024-03-29T11:44:21.516 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.505+0000 7f25c2eff640 10 monclient: build_initial_monmap 2024-03-29T11:44:21.516 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.505+0000 7f25c2eff640 10 monclient: monmap: 2024-03-29T11:44:21.516 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:44:21.516 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:21.516 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:44:21.516 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:44:21.516 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:21.516 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:21.516 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:44:21.516 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:44:21.516 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:44:21.516 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:21.524 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.505+0000 7f25c2eff640 1 librados: starting msgr at 2024-03-29T11:44:21.525 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.505+0000 7f25c2eff640 1 librados: starting objecter 2024-03-29T11:44:21.525 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.505+0000 7f25c2eff640 1 Processor -- start 2024-03-29T11:44:21.525 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.505+0000 7f25c2eff640 1 -- start start 2024-03-29T11:44:21.525 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.505+0000 7f25c2eff640 1 librados: setting wanted keys 2024-03-29T11:44:21.525 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.505+0000 7f25c2eff640 1 librados: calling monclient init 2024-03-29T11:44:21.525 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.505+0000 7f25c2eff640 10 monclient: init 2024-03-29T11:44:21.525 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.506+0000 7f25c2eff640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:21.525 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.506+0000 7f25c2eff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:44:21.525 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.506+0000 7f25c2eff640 10 monclient(hunting): picked mon.noname-c con 0x7f25bc071a60 addr v1:172.21.15.72:6790/0 2024-03-29T11:44:21.525 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.506+0000 7f25c2eff640 10 monclient(hunting): picked mon.noname-a con 0x7f25bc0b4890 addr v1:172.21.15.72:6789/0 2024-03-29T11:44:21.525 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.506+0000 7f25c2eff640 10 monclient(hunting): picked mon.noname-b con 0x7f25bc152120 addr v1:172.21.15.89:6789/0 2024-03-29T11:44:21.525 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.506+0000 7f25c2eff640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25bc0adb30 con 0x7f25bc0b4890 2024-03-29T11:44:21.525 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.506+0000 7f25c2eff640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25bc0ac2e0 con 0x7f25bc152120 2024-03-29T11:44:21.525 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.506+0000 7f25c2eff640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25bc148e40 con 0x7f25bc071a60 2024-03-29T11:44:21.526 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.506+0000 7f25c2eff640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:21.526 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.506+0000 7f25c2eff640 10 monclient(hunting): authenticate will time out at 1420.795834s 2024-03-29T11:44:21.526 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.509+0000 7f25c0c9b640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f25bc071a60 0x7f25bc148730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:48284/0 (socket says 172.21.15.72:48284) 2024-03-29T11:44:21.526 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.509+0000 7f25c0c9b640 1 -- 172.21.15.72:0/1188881645 learned_addr learned my addr 172.21.15.72:0/1188881645 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:44:21.526 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.511+0000 7f25b97fa640 10 client.?.objecter ms_handle_connect 0x7f25bc071a60 2024-03-29T11:44:21.526 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.511+0000 7f25b97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:21.526 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.511+0000 7f25b97fa640 10 client.?.objecter ms_handle_connect 0x7f25bc152120 2024-03-29T11:44:21.526 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.511+0000 7f25b97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:21.526 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.511+0000 7f25b97fa640 1 -- 172.21.15.72:0/1188881645 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4097859985 0 0) 0x7f25bc148e40 con 0x7f25bc071a60 2024-03-29T11:44:21.526 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.511+0000 7f25b97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:44:21.526 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.511+0000 7f25b97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:21.526 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.511+0000 7f25b97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:21.526 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.511+0000 7f25b97fa640 10 monclient(hunting): my global_id is 4577 2024-03-29T11:44:21.526 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.511+0000 7f25b97fa640 1 -- 172.21.15.72:0/1188881645 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25a8004eb0 con 0x7f25bc071a60 2024-03-29T11:44:21.527 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.511+0000 7f25b97fa640 1 -- 172.21.15.72:0/1188881645 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 946630908 0 0) 0x7f25bc0ac2e0 con 0x7f25bc152120 2024-03-29T11:44:21.527 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.511+0000 7f25b97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:44:21.527 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.511+0000 7f25b97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:21.527 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.511+0000 7f25b97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:21.527 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.512+0000 7f25b97fa640 10 monclient(hunting): my global_id is 4504 2024-03-29T11:44:21.527 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.512+0000 7f25b97fa640 1 -- 172.21.15.72:0/1188881645 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25bc148e40 con 0x7f25bc152120 2024-03-29T11:44:21.527 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.512+0000 7f25b97fa640 1 -- 172.21.15.72:0/1188881645 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1778074196 0 0) 0x7f25a8004eb0 con 0x7f25bc071a60 2024-03-29T11:44:21.527 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.512+0000 7f25b97fa640 1 -- 172.21.15.72:0/1188881645 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25bc0ac2e0 con 0x7f25bc071a60 2024-03-29T11:44:21.527 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.512+0000 7f25b97fa640 1 -- 172.21.15.72:0/1188881645 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f25b4001d90 con 0x7f25bc071a60 2024-03-29T11:44:21.527 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.512+0000 7f25b97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:21.527 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.512+0000 7f25b97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:44:21.527 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.512+0000 7f25b97fa640 10 monclient(hunting): dump: 2024-03-29T11:44:21.527 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:44:21.527 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:44:21.527 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:21.527 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:21.528 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:21.528 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:21.528 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:44:21.528 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:44:21.528 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:44:21.528 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:21.528 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.512+0000 7f25b97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:21.528 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.512+0000 7f25b97fa640 1 -- 172.21.15.72:0/1188881645 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3540463998 0 0) 0x7f25bc148e40 con 0x7f25bc152120 2024-03-29T11:44:21.528 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.512+0000 7f25b97fa640 1 -- 172.21.15.72:0/1188881645 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25a8004eb0 con 0x7f25bc152120 2024-03-29T11:44:21.528 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.512+0000 7f25b97fa640 1 -- 172.21.15.72:0/1188881645 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f25ac0016e0 con 0x7f25bc152120 2024-03-29T11:44:21.530 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.512+0000 7f25b97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:21.530 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.512+0000 7f25b97fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:44:21.531 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.512+0000 7f25b97fa640 10 monclient(hunting): dump: 2024-03-29T11:44:21.531 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:44:21.531 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:44:21.531 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:21.531 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:21.531 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:21.531 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:21.531 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:44:21.531 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:44:21.532 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:44:21.532 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:21.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.512+0000 7f25b97fa640 1 -- 172.21.15.72:0/1188881645 <== mon.2 v1:172.21.15.72:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2009311374 0 0) 0x7f25bc0ac2e0 con 0x7f25bc071a60 2024-03-29T11:44:21.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.512+0000 7f25b97fa640 1 -- 172.21.15.72:0/1188881645 >> v1:172.21.15.89:6789/0 conn(0x7f25bc152120 legacy=0x7f25bc200290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:21.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.512+0000 7f25b97fa640 1 -- 172.21.15.72:0/1188881645 >> v1:172.21.15.72:6789/0 conn(0x7f25bc0b4890 legacy=0x7f25bc14be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:21.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.512+0000 7f25b97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:44:21.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.512+0000 7f25b97fa640 1 monclient: found mon.c 2024-03-29T11:44:21.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.512+0000 7f25b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:21.532 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.512+0000 7f25b97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:44:21.533 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.512+0000 7f25b97fa640 1 -- 172.21.15.72:0/1188881645 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25bc148fd0 con 0x7f25bc071a60 2024-03-29T11:44:21.533 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.513+0000 7f25b97fa640 10 monclient: _finish_auth 0 2024-03-29T11:44:21.533 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.513+0000 7f25b97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:44:21.533 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.513+0000 7f25b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:21.533 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.513+0000 7f25c2eff640 5 monclient: authenticate success, global_id 4577 2024-03-29T11:44:21.533 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.513+0000 7f25c2eff640 10 monclient: _renew_subs 2024-03-29T11:44:21.533 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.513+0000 7f25c2eff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:44:21.533 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.513+0000 7f25c2eff640 1 -- 172.21.15.72:0/1188881645 --> v1:172.21.15.72:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25bc142990 con 0x7f25bc071a60 2024-03-29T11:44:21.533 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.513+0000 7f25c2eff640 10 client.4577.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:21.534 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.513+0000 7f25c2eff640 10 monclient: _renew_subs 2024-03-29T11:44:21.534 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.513+0000 7f25c2eff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:44:21.534 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.513+0000 7f25c2eff640 1 -- 172.21.15.72:0/1188881645 --> v1:172.21.15.72:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f25bc14c740 con 0x7f25bc071a60 2024-03-29T11:44:21.534 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.513+0000 7f25c2eff640 1 librados: init done 2024-03-29T11:44:21.534 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.516+0000 7f25b97fa640 1 -- 172.21.15.72:0/1188881645 <== mon.2 v1:172.21.15.72:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f25b4001fd0 con 0x7f25bc071a60 2024-03-29T11:44:21.534 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.516+0000 7f25b97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:21.534 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.516+0000 7f25b97fa640 1 -- 172.21.15.72:0/1188881645 <== mon.2 v1:172.21.15.72:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f25b4003de0 con 0x7f25bc071a60 2024-03-29T11:44:21.534 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.516+0000 7f25b97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:21.534 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.516+0000 7f25b97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:21.535 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.516+0000 7f25b97fa640 10 monclient: dump: 2024-03-29T11:44:21.535 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:44:21.535 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:44:21.535 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:21.535 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:21.535 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:21.535 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:21.535 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:44:21.535 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:44:21.536 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:44:21.536 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:21.536 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.516+0000 7f25b97fa640 1 -- 172.21.15.72:0/1188881645 <== mon.2 v1:172.21.15.72:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f25b4004000 con 0x7f25bc071a60 2024-03-29T11:44:21.536 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.516+0000 7f25ba7fc640 4 set_mon_vals no callback set 2024-03-29T11:44:21.536 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.516+0000 7f25b97fa640 1 -- 172.21.15.72:0/1188881645 <== mon.2 v1:172.21.15.72:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f25b4084ef0 con 0x7f25bc071a60 2024-03-29T11:44:21.536 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.516+0000 7f25b97fa640 10 client.4577.objecter ms_dispatch 0x7f25bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:21.536 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.516+0000 7f25b97fa640 3 client.4577.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:21.536 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.516+0000 7f25b97fa640 3 client.4577.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:21.537 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.516+0000 7f25c2eff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:21.537 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.517+0000 7f25c2eff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:21.537 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.517+0000 7f25c2eff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:44:21.537 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.517+0000 7f25c2eff640 1 -- 172.21.15.72:0/1188881645 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f259c005080 con 0x7f25bc071a60 2024-03-29T11:44:21.537 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.517+0000 7f25b97fa640 20 client.4577.objecter dump_active .. 0 homeless 2024-03-29T11:44:21.537 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.517+0000 7f25bbfff640 10 monclient: get_auth_request con 0x7f25a806dfa0 auth_method 0 2024-03-29T11:44:21.537 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.518+0000 7f25b97fa640 10 client.4577.objecter ms_handle_connect 0x7f25a806dfa0 2024-03-29T11:44:21.538 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.525+0000 7f25b97fa640 1 -- 172.21.15.72:0/1188881645 <== mon.2 v1:172.21.15.72:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f25b4050850 con 0x7f25bc071a60 2024-03-29T11:44:21.538 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.525+0000 7f25b97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:21.538 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.525+0000 7f25b97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:21.851 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.846+0000 7f25c2eff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:21.851 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.846+0000 7f25c2eff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:21.851 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.846+0000 7f25c2eff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:44:21.851 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.846+0000 7f25c2eff640 1 -- 172.21.15.72:0/1188881645 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f259c005330 con 0x7f25bc071a60 2024-03-29T11:44:21.852 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.847+0000 7f25b97fa640 1 -- 172.21.15.72:0/1188881645 <== mon.2 v1:172.21.15.72:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f25b4052780 con 0x7f25bc071a60 2024-03-29T11:44:21.852 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.847+0000 7f25b97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:21.852 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.847+0000 7f25b97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:21.853 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:44:21.853 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:21.857 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.851+0000 7f25c2eff640 10 librados: watch_flush enter 2024-03-29T11:44:21.857 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.851+0000 7f25c2eff640 10 librados: watch_flush exit 2024-03-29T11:44:21.857 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.851+0000 7f25c2eff640 20 client.4577.objecter shutdown clearing up homeless session... 2024-03-29T11:44:21.857 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.851+0000 7f25c2eff640 10 client.4577.objecter successfully canceled tick 2024-03-29T11:44:21.859 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.852+0000 7f25c2eff640 1 -- 172.21.15.72:0/1188881645 >> v1:172.21.15.72:6816/1578866102 conn(0x7f25a806dfa0 legacy=0x7f25a8070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:21.859 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.852+0000 7f25c2eff640 10 monclient: shutdown 2024-03-29T11:44:21.859 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.852+0000 7f25c2eff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:21.859 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.852+0000 7f25c2eff640 1 -- 172.21.15.72:0/1188881645 >> v1:172.21.15.72:6790/0 conn(0x7f25bc071a60 legacy=0x7f25bc148730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:21.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.852+0000 7f25c2eff640 1 -- 172.21.15.72:0/1188881645 shutdown_connections 2024-03-29T11:44:21.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.852+0000 7f25c2eff640 1 -- 172.21.15.72:0/1188881645 >> 172.21.15.72:0/1188881645 conn(0x7f25bc066a00 msgr2=0x7f25bc0b3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:21.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.853+0000 7f25c2eff640 1 -- 172.21.15.72:0/1188881645 shutdown_connections 2024-03-29T11:44:21.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.853+0000 7f25c2eff640 1 -- 172.21.15.72:0/1188881645 wait complete. 2024-03-29T11:44:21.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:21.853+0000 7f25c2eff640 1 librados: shutdown 2024-03-29T11:44:21.875 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:28.681 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:28.842 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.838+0000 7f95ca88f640 10 monclient: get_monmap_and_config 2024-03-29T11:44:28.842 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.838+0000 7f95ca88f640 10 monclient: build_initial_monmap 2024-03-29T11:44:28.842 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.838+0000 7f95ca88f640 10 monclient: monmap: 2024-03-29T11:44:28.842 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:44:28.842 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:44:28.842 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:44:28.839170+0000 2024-03-29T11:44:28.842 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:44:28.839170+0000 2024-03-29T11:44:28.842 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:28.842 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:28.842 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:44:28.843 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:44:28.843 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:44:28.843 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:28.843 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.839+0000 7f95ca88f640 1 Processor -- start 2024-03-29T11:44:28.844 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.840+0000 7f95ca88f640 1 -- start start 2024-03-29T11:44:28.844 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.840+0000 7f95ca88f640 10 monclient: init 2024-03-29T11:44:28.844 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.841+0000 7f95ca88f640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:28.845 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.841+0000 7f95ca88f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:44:28.845 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.841+0000 7f95ca88f640 10 monclient(hunting): picked mon.noname-a con 0x7f95bc038c60 addr v1:172.21.15.72:6789/0 2024-03-29T11:44:28.845 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.841+0000 7f95ca88f640 10 monclient(hunting): picked mon.noname-c con 0x7f95bc0324b0 addr v1:172.21.15.89:6789/0 2024-03-29T11:44:28.845 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.841+0000 7f95c988d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f95bc038c60 0x7f95bc0c4ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60584/0 (socket says 172.21.15.72:60584) 2024-03-29T11:44:28.845 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.841+0000 7f95ca88f640 10 monclient(hunting): picked mon.noname-b con 0x7f95bc0d11e0 addr v1:172.21.15.72:6790/0 2024-03-29T11:44:28.845 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.841+0000 7f95ca88f640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95bc0caf80 con 0x7f95bc038c60 2024-03-29T11:44:28.845 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.841+0000 7f95c988d640 1 -- 172.21.15.72:0/122366188 learned_addr learned my addr 172.21.15.72:0/122366188 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:44:28.846 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.841+0000 7f95ca88f640 1 -- 172.21.15.72:0/122366188 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95bc039850 con 0x7f95bc0324b0 2024-03-29T11:44:28.846 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.841+0000 7f95ca88f640 1 -- 172.21.15.72:0/122366188 --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95bc0c78d0 con 0x7f95bc0d11e0 2024-03-29T11:44:28.846 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.841+0000 7f95ca88f640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:28.846 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.841+0000 7f95ca88f640 10 monclient(hunting): authenticate will time out at 1428.129670s 2024-03-29T11:44:28.848 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.842+0000 7f95c888b640 1 -- 172.21.15.72:0/122366188 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4005941148 0 0) 0x7f95bc0caf80 con 0x7f95bc038c60 2024-03-29T11:44:28.848 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.842+0000 7f95c888b640 10 monclient(hunting): _negotiate 2024-03-29T11:44:28.848 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.842+0000 7f95c888b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:28.848 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.842+0000 7f95c888b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:28.848 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.842+0000 7f95c888b640 10 monclient(hunting): my global_id is 4551 2024-03-29T11:44:28.848 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.842+0000 7f95c888b640 1 -- 172.21.15.72:0/122366188 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95b0003520 con 0x7f95bc038c60 2024-03-29T11:44:28.848 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.842+0000 7f95c888b640 1 -- 172.21.15.72:0/122366188 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2555866586 0 0) 0x7f95bc039850 con 0x7f95bc0324b0 2024-03-29T11:44:28.848 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.842+0000 7f95c888b640 10 monclient(hunting): _negotiate 2024-03-29T11:44:28.848 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.842+0000 7f95c888b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:28.848 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.842+0000 7f95c888b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:28.848 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.842+0000 7f95c888b640 10 monclient(hunting): my global_id is 4507 2024-03-29T11:44:28.848 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.842+0000 7f95c888b640 1 -- 172.21.15.72:0/122366188 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95bc0caf80 con 0x7f95bc0324b0 2024-03-29T11:44:28.848 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.843+0000 7f95c888b640 1 -- 172.21.15.72:0/122366188 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1547885629 0 0) 0x7f95bc0c78d0 con 0x7f95bc0d11e0 2024-03-29T11:44:28.848 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.843+0000 7f95c888b640 10 monclient(hunting): _negotiate 2024-03-29T11:44:28.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.843+0000 7f95c888b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:28.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.843+0000 7f95c888b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:28.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.843+0000 7f95c888b640 10 monclient(hunting): my global_id is 4580 2024-03-29T11:44:28.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.843+0000 7f95c888b640 1 -- 172.21.15.72:0/122366188 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95bc039850 con 0x7f95bc0d11e0 2024-03-29T11:44:28.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.843+0000 7f95c888b640 1 -- 172.21.15.72:0/122366188 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 142038306 0 0) 0x7f95b0003520 con 0x7f95bc038c60 2024-03-29T11:44:28.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.843+0000 7f95c888b640 1 -- 172.21.15.72:0/122366188 >> v1:172.21.15.72:6790/0 conn(0x7f95bc0d11e0 legacy=0x7f95bc0d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:28.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.843+0000 7f95c888b640 1 -- 172.21.15.72:0/122366188 >> v1:172.21.15.89:6789/0 conn(0x7f95bc0324b0 legacy=0x7f95bc0cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:28.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.843+0000 7f95c888b640 10 monclient: _finish_hunting 0 2024-03-29T11:44:28.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.843+0000 7f95c888b640 1 monclient: found mon.noname-a 2024-03-29T11:44:28.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.843+0000 7f95c888b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:28.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.843+0000 7f95c888b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.72:6789/0 2024-03-29T11:44:28.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.843+0000 7f95c888b640 1 -- 172.21.15.72:0/122366188 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95bc0c7a60 con 0x7f95bc038c60 2024-03-29T11:44:28.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.843+0000 7f95c888b640 10 monclient: _finish_auth 0 2024-03-29T11:44:28.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.843+0000 7f95c888b640 10 monclient: _check_auth_tickets 2024-03-29T11:44:28.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.843+0000 7f95c888b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:28.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.843+0000 7f95c888b640 1 -- 172.21.15.72:0/122366188 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f95c404f730 con 0x7f95bc038c60 2024-03-29T11:44:28.850 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.843+0000 7f95ca88f640 5 monclient: authenticate success, global_id 4551 2024-03-29T11:44:28.850 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.843+0000 7f95ca88f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:28.850 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.843+0000 7f95c888b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:28.850 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.843+0000 7f95c888b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:44:28.850 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.843+0000 7f95c888b640 10 monclient: dump: 2024-03-29T11:44:28.850 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:44:28.850 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:44:28.850 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:28.850 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:28.850 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:28.850 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:28.850 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:44:28.850 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:44:28.850 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:44:28.850 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:28.851 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.843+0000 7f95ca88f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:28.851 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.843+0000 7f95c888b640 1 -- 172.21.15.72:0/122366188 <== mon.0 v1:172.21.15.72:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95c40008d0 con 0x7f95bc038c60 2024-03-29T11:44:28.851 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.843+0000 7f95c888b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:28.851 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.843+0000 7f95c888b640 1 -- 172.21.15.72:0/122366188 <== mon.0 v1:172.21.15.72:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f95c40758b0 con 0x7f95bc038c60 2024-03-29T11:44:28.851 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.843+0000 7f95c888b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:28.851 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.843+0000 7f95c888b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:28.851 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.843+0000 7f95c888b640 10 monclient: dump: 2024-03-29T11:44:28.851 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:44:28.851 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:44:28.851 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:28.851 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:28.851 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:28.851 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:28.851 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:44:28.851 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:44:28.851 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:44:28.851 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:28.852 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.843+0000 7f95ca88f640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:28.852 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.843+0000 7f95ca88f640 4 set_mon_vals no callback set 2024-03-29T11:44:28.857 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.844+0000 7f95ca88f640 10 monclient: shutdown 2024-03-29T11:44:28.857 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.844+0000 7f95ca88f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:28.857 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.844+0000 7f95ca88f640 1 -- 172.21.15.72:0/122366188 >> v1:172.21.15.72:6789/0 conn(0x7f95bc038c60 legacy=0x7f95bc0c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:28.858 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.845+0000 7f95ca88f640 1 -- 172.21.15.72:0/122366188 shutdown_connections 2024-03-29T11:44:28.858 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.845+0000 7f95ca88f640 1 -- 172.21.15.72:0/122366188 >> 172.21.15.72:0/122366188 conn(0x7f95bc005c20 msgr2=0x7f95bc006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:28.858 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.845+0000 7f95ca88f640 1 -- 172.21.15.72:0/122366188 shutdown_connections 2024-03-29T11:44:28.858 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.845+0000 7f95ca88f640 1 -- 172.21.15.72:0/122366188 wait complete. 2024-03-29T11:44:28.858 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.846+0000 7f95ca88f640 10 monclient: build_initial_monmap 2024-03-29T11:44:28.858 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.846+0000 7f95ca88f640 10 monclient: monmap: 2024-03-29T11:44:28.858 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:44:28.858 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:28.858 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:44:28.858 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:44:28.858 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:28.858 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:28.858 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:44:28.858 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:44:28.858 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:44:28.859 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:28.859 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.846+0000 7f95ca88f640 1 librados: starting msgr at 2024-03-29T11:44:28.859 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.846+0000 7f95ca88f640 1 librados: starting objecter 2024-03-29T11:44:28.859 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.846+0000 7f95ca88f640 1 Processor -- start 2024-03-29T11:44:28.859 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.847+0000 7f95ca88f640 1 -- start start 2024-03-29T11:44:28.859 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.847+0000 7f95ca88f640 1 librados: setting wanted keys 2024-03-29T11:44:28.859 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.847+0000 7f95ca88f640 1 librados: calling monclient init 2024-03-29T11:44:28.859 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.847+0000 7f95ca88f640 10 monclient: init 2024-03-29T11:44:28.859 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.847+0000 7f95ca88f640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:28.859 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.847+0000 7f95ca88f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:44:28.859 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.847+0000 7f95ca88f640 10 monclient(hunting): picked mon.noname-b con 0x7f95bc0324b0 addr v1:172.21.15.89:6789/0 2024-03-29T11:44:28.859 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.848+0000 7f95ca88f640 10 monclient(hunting): picked mon.noname-c con 0x7f95bc038c60 addr v1:172.21.15.72:6790/0 2024-03-29T11:44:28.859 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.848+0000 7f95c908c640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f95bc038c60 0x7f95bc034ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:50538/0 (socket says 172.21.15.72:50538) 2024-03-29T11:44:28.859 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.848+0000 7f95c908c640 1 -- 172.21.15.72:0/1582297549 learned_addr learned my addr 172.21.15.72:0/1582297549 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:44:28.859 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.848+0000 7f95ca88f640 10 monclient(hunting): picked mon.noname-a con 0x7f95bc0d11e0 addr v1:172.21.15.72:6789/0 2024-03-29T11:44:28.859 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.848+0000 7f95ca88f640 1 -- 172.21.15.72:0/1582297549 --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95bc02d170 con 0x7f95bc0d11e0 2024-03-29T11:44:28.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.848+0000 7f95ca88f640 1 -- 172.21.15.72:0/1582297549 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95bc030360 con 0x7f95bc0324b0 2024-03-29T11:44:28.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.848+0000 7f95ca88f640 1 -- 172.21.15.72:0/1582297549 --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95bc0bcf40 con 0x7f95bc038c60 2024-03-29T11:44:28.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.848+0000 7f95ca88f640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:28.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.848+0000 7f95ca88f640 10 monclient(hunting): authenticate will time out at 1428.136419s 2024-03-29T11:44:28.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.848+0000 7f95ba7fc640 10 client.?.objecter ms_handle_connect 0x7f95bc038c60 2024-03-29T11:44:28.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.848+0000 7f95ba7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:28.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.848+0000 7f95ba7fc640 10 client.?.objecter ms_handle_connect 0x7f95bc0d11e0 2024-03-29T11:44:28.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.848+0000 7f95ba7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:28.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.848+0000 7f95ba7fc640 10 client.?.objecter ms_handle_connect 0x7f95bc0324b0 2024-03-29T11:44:28.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.848+0000 7f95ba7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:28.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.848+0000 7f95ba7fc640 1 -- 172.21.15.72:0/1582297549 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2425950139 0 0) 0x7f95bc0bcf40 con 0x7f95bc038c60 2024-03-29T11:44:28.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.848+0000 7f95ba7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:44:28.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.848+0000 7f95ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:28.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.848+0000 7f95ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:28.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.848+0000 7f95ba7fc640 10 monclient(hunting): my global_id is 4583 2024-03-29T11:44:28.861 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.848+0000 7f95ba7fc640 1 -- 172.21.15.72:0/1582297549 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95a8004eb0 con 0x7f95bc038c60 2024-03-29T11:44:28.861 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.849+0000 7f95ba7fc640 1 -- 172.21.15.72:0/1582297549 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 187555157 0 0) 0x7f95bc030360 con 0x7f95bc0324b0 2024-03-29T11:44:28.861 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.849+0000 7f95ba7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:44:28.861 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.849+0000 7f95ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:28.861 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.849+0000 7f95ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:28.861 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.849+0000 7f95ba7fc640 10 monclient(hunting): my global_id is 4510 2024-03-29T11:44:28.861 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.849+0000 7f95ba7fc640 1 -- 172.21.15.72:0/1582297549 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95bc0bcf40 con 0x7f95bc0324b0 2024-03-29T11:44:28.861 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.849+0000 7f95ba7fc640 1 -- 172.21.15.72:0/1582297549 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 729998004 0 0) 0x7f95bc02d170 con 0x7f95bc0d11e0 2024-03-29T11:44:28.861 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.849+0000 7f95ba7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:44:28.861 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.849+0000 7f95ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:28.861 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.849+0000 7f95ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:28.861 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.849+0000 7f95ba7fc640 10 monclient(hunting): my global_id is 4554 2024-03-29T11:44:28.861 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.849+0000 7f95ba7fc640 1 -- 172.21.15.72:0/1582297549 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95bc030360 con 0x7f95bc0d11e0 2024-03-29T11:44:28.861 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.849+0000 7f95ba7fc640 1 -- 172.21.15.72:0/1582297549 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2226536754 0 0) 0x7f95bc0bcf40 con 0x7f95bc0324b0 2024-03-29T11:44:28.861 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.849+0000 7f95ba7fc640 1 -- 172.21.15.72:0/1582297549 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95bc02d170 con 0x7f95bc0324b0 2024-03-29T11:44:28.861 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.849+0000 7f95ba7fc640 1 -- 172.21.15.72:0/1582297549 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2673901261 0 0) 0x7f95bc030360 con 0x7f95bc0d11e0 2024-03-29T11:44:28.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.849+0000 7f95ba7fc640 1 -- 172.21.15.72:0/1582297549 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95bc0bcf40 con 0x7f95bc0d11e0 2024-03-29T11:44:28.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.849+0000 7f95ba7fc640 1 -- 172.21.15.72:0/1582297549 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f95c404f400 con 0x7f95bc0324b0 2024-03-29T11:44:28.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.849+0000 7f95ba7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:28.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.849+0000 7f95ba7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:28.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.849+0000 7f95ba7fc640 10 monclient(hunting): dump: 2024-03-29T11:44:28.862 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:44:28.862 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:44:28.862 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:28.862 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:28.862 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:28.862 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:28.862 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:44:28.862 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:44:28.862 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:44:28.862 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:28.863 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.849+0000 7f95ba7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:28.863 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.849+0000 7f95ba7fc640 1 -- 172.21.15.72:0/1582297549 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f95b4001790 con 0x7f95bc0d11e0 2024-03-29T11:44:28.863 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.849+0000 7f95ba7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:28.863 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.849+0000 7f95ba7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:28.863 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.849+0000 7f95ba7fc640 10 monclient(hunting): dump: 2024-03-29T11:44:28.863 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:44:28.863 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:44:28.863 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:28.863 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:28.863 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:28.863 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:28.863 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:44:28.863 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:44:28.863 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:44:28.863 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:28.864 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.849+0000 7f95ba7fc640 1 -- 172.21.15.72:0/1582297549 <== mon.1 v1:172.21.15.89:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2950677572 0 0) 0x7f95bc02d170 con 0x7f95bc0324b0 2024-03-29T11:44:28.864 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.850+0000 7f95ba7fc640 1 -- 172.21.15.72:0/1582297549 >> v1:172.21.15.72:6790/0 conn(0x7f95bc038c60 legacy=0x7f95bc034ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:28.864 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.850+0000 7f95ba7fc640 1 -- 172.21.15.72:0/1582297549 >> v1:172.21.15.72:6789/0 conn(0x7f95bc0d11e0 legacy=0x7f95bc163580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:28.864 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.850+0000 7f95ba7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:44:28.864 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.850+0000 7f95ba7fc640 1 monclient: found mon.b 2024-03-29T11:44:28.864 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.850+0000 7f95ba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:28.864 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.850+0000 7f95ba7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:44:28.864 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.850+0000 7f95ba7fc640 1 -- 172.21.15.72:0/1582297549 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95bc0bd220 con 0x7f95bc0324b0 2024-03-29T11:44:28.864 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.850+0000 7f95ba7fc640 10 monclient: _finish_auth 0 2024-03-29T11:44:28.864 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.850+0000 7f95ba7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:44:28.864 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.850+0000 7f95ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:28.864 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.850+0000 7f95ca88f640 5 monclient: authenticate success, global_id 4510 2024-03-29T11:44:28.864 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.850+0000 7f95ca88f640 10 monclient: _renew_subs 2024-03-29T11:44:28.864 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.850+0000 7f95ca88f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:44:28.865 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.850+0000 7f95ca88f640 1 -- 172.21.15.72:0/1582297549 --> v1:172.21.15.89:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95bc0bd500 con 0x7f95bc0324b0 2024-03-29T11:44:28.865 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.850+0000 7f95ca88f640 10 client.4510.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:28.865 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.850+0000 7f95ca88f640 10 monclient: _renew_subs 2024-03-29T11:44:28.865 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.850+0000 7f95ca88f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:44:28.865 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.850+0000 7f95ca88f640 1 -- 172.21.15.72:0/1582297549 --> v1:172.21.15.89:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f95bc0bb1c0 con 0x7f95bc0324b0 2024-03-29T11:44:28.865 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.850+0000 7f95ca88f640 1 librados: init done 2024-03-29T11:44:28.865 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.850+0000 7f95ba7fc640 1 -- 172.21.15.72:0/1582297549 <== mon.1 v1:172.21.15.89:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95c404f730 con 0x7f95bc0324b0 2024-03-29T11:44:28.865 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.850+0000 7f95ba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:28.865 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.850+0000 7f95ba7fc640 1 -- 172.21.15.72:0/1582297549 <== mon.1 v1:172.21.15.89:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f95c4067d90 con 0x7f95bc0324b0 2024-03-29T11:44:28.865 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.850+0000 7f95ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:28.865 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.850+0000 7f95bbfff640 4 set_mon_vals no callback set 2024-03-29T11:44:28.865 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.850+0000 7f95ba7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:44:28.865 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.850+0000 7f95ba7fc640 10 monclient: dump: 2024-03-29T11:44:28.865 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:44:28.865 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:44:28.865 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:28.866 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:28.866 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:28.866 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:28.866 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:44:28.866 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:44:28.866 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:44:28.866 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:28.866 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.853+0000 7f95ba7fc640 1 -- 172.21.15.72:0/1582297549 <== mon.1 v1:172.21.15.89:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f95c4067ef0 con 0x7f95bc0324b0 2024-03-29T11:44:28.867 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.854+0000 7f95c908c640 10 monclient: get_auth_request con 0x7f95a806de80 auth_method 0 2024-03-29T11:44:28.867 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.854+0000 7f95ca88f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:28.867 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.854+0000 7f95ca88f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:28.867 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.854+0000 7f95ca88f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:44:28.867 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.855+0000 7f95ba7fc640 10 client.4510.objecter ms_handle_connect 0x7f95a806de80 2024-03-29T11:44:28.868 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.855+0000 7f95ba7fc640 1 -- 172.21.15.72:0/1582297549 <== mon.1 v1:172.21.15.89:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f95c40ff1c0 con 0x7f95bc0324b0 2024-03-29T11:44:28.868 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.855+0000 7f95ba7fc640 10 client.4510.objecter ms_dispatch 0x7f95c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:28.868 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.855+0000 7f95ba7fc640 3 client.4510.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:28.868 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.855+0000 7f95ba7fc640 3 client.4510.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:28.868 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.855+0000 7f95ba7fc640 20 client.4510.objecter dump_active .. 0 homeless 2024-03-29T11:44:28.868 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.854+0000 7f95ca88f640 1 -- 172.21.15.72:0/1582297549 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95ac005080 con 0x7f95bc0324b0 2024-03-29T11:44:28.868 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.862+0000 7f95ba7fc640 1 -- 172.21.15.72:0/1582297549 <== mon.1 v1:172.21.15.89:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f95c40caaa0 con 0x7f95bc0324b0 2024-03-29T11:44:28.868 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.862+0000 7f95ba7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:28.868 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:28.862+0000 7f95ba7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:29.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:29.222+0000 7f95ca88f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:29.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:29.222+0000 7f95ca88f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:29.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:29.222+0000 7f95ca88f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:44:29.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:29.222+0000 7f95ca88f640 1 -- 172.21.15.72:0/1582297549 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f95ac005330 con 0x7f95bc0324b0 2024-03-29T11:44:29.227 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:29.223+0000 7f95ba7fc640 1 -- 172.21.15.72:0/1582297549 <== mon.1 v1:172.21.15.89:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f95c40cc9d0 con 0x7f95bc0324b0 2024-03-29T11:44:29.227 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:29.223+0000 7f95ba7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:29.227 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:29.223+0000 7f95ba7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:29.227 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:44:29.228 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:29.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:29.226+0000 7f95ca88f640 10 librados: watch_flush enter 2024-03-29T11:44:29.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:29.226+0000 7f95ca88f640 10 librados: watch_flush exit 2024-03-29T11:44:29.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:29.226+0000 7f95ca88f640 20 client.4510.objecter shutdown clearing up homeless session... 2024-03-29T11:44:29.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:29.226+0000 7f95ca88f640 10 client.4510.objecter successfully canceled tick 2024-03-29T11:44:29.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:29.227+0000 7f95ca88f640 1 -- 172.21.15.72:0/1582297549 >> v1:172.21.15.72:6816/1578866102 conn(0x7f95a806de80 legacy=0x7f95a8070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:29.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:29.227+0000 7f95ca88f640 10 monclient: shutdown 2024-03-29T11:44:29.231 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:29.227+0000 7f95ca88f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:29.232 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:29.227+0000 7f95ca88f640 1 -- 172.21.15.72:0/1582297549 >> v1:172.21.15.89:6789/0 conn(0x7f95bc0324b0 legacy=0x7f95bc0bbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:29.232 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:29.227+0000 7f95ca88f640 1 -- 172.21.15.72:0/1582297549 shutdown_connections 2024-03-29T11:44:29.232 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:29.227+0000 7f95ca88f640 1 -- 172.21.15.72:0/1582297549 >> 172.21.15.72:0/1582297549 conn(0x7f95bc005c20 msgr2=0x7f95bc0043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:29.232 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:29.227+0000 7f95ca88f640 1 -- 172.21.15.72:0/1582297549 shutdown_connections 2024-03-29T11:44:29.232 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:29.227+0000 7f95ca88f640 1 -- 172.21.15.72:0/1582297549 wait complete. 2024-03-29T11:44:29.232 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:29.227+0000 7f95ca88f640 1 librados: shutdown 2024-03-29T11:44:29.242 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:36.051 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:36.202 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.197+0000 7faf06979640 10 monclient: get_monmap_and_config 2024-03-29T11:44:36.202 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.197+0000 7faf06979640 10 monclient: build_initial_monmap 2024-03-29T11:44:36.203 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.197+0000 7faf06979640 10 monclient: monmap: 2024-03-29T11:44:36.203 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:44:36.203 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:44:36.203 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:44:36.198621+0000 2024-03-29T11:44:36.203 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:44:36.198621+0000 2024-03-29T11:44:36.203 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:36.204 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:36.204 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:44:36.204 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:44:36.204 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:44:36.204 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:36.209 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.200+0000 7faf06979640 1 Processor -- start 2024-03-29T11:44:36.209 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.200+0000 7faf06979640 1 -- start start 2024-03-29T11:44:36.210 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.200+0000 7faf06979640 10 monclient: init 2024-03-29T11:44:36.210 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.201+0000 7faf06979640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:36.210 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.201+0000 7faf06979640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:44:36.210 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.201+0000 7faf06979640 10 monclient(hunting): picked mon.noname-c con 0x7faf000b0860 addr v1:172.21.15.89:6789/0 2024-03-29T11:44:36.210 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.201+0000 7faf06979640 10 monclient(hunting): picked mon.noname-b con 0x7faf0013ef30 addr v1:172.21.15.72:6790/0 2024-03-29T11:44:36.211 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.201+0000 7faf06979640 10 monclient(hunting): picked mon.noname-a con 0x7faf000af8e0 addr v1:172.21.15.72:6789/0 2024-03-29T11:44:36.211 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.201+0000 7faf06979640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf000b0c40 con 0x7faf000af8e0 2024-03-29T11:44:36.211 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.201+0000 7faf06979640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf000aed10 con 0x7faf000b0860 2024-03-29T11:44:36.211 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.201+0000 7faf06979640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf000afd50 con 0x7faf0013ef30 2024-03-29T11:44:36.211 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.201+0000 7faf06979640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:36.212 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.201+0000 7faf06979640 10 monclient(hunting): authenticate will time out at 1435.489513s 2024-03-29T11:44:36.213 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.206+0000 7faefffff640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7faf000b0860 0x7faf0013e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:40577/0 (socket says 172.21.15.72:40577) 2024-03-29T11:44:36.213 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.206+0000 7faefffff640 1 -- 172.21.15.72:0/2627253376 learned_addr learned my addr 172.21.15.72:0/2627253376 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:44:36.216 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.209+0000 7faefeffd640 1 -- 172.21.15.72:0/2627253376 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2577792887 0 0) 0x7faf000b0c40 con 0x7faf000af8e0 2024-03-29T11:44:36.216 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.209+0000 7faefeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:44:36.216 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.209+0000 7faefeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:36.217 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.209+0000 7faefeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:36.217 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.209+0000 7faefeffd640 10 monclient(hunting): my global_id is 4557 2024-03-29T11:44:36.217 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.209+0000 7faefeffd640 1 -- 172.21.15.72:0/2627253376 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faee8003520 con 0x7faf000af8e0 2024-03-29T11:44:36.217 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.209+0000 7faefeffd640 1 -- 172.21.15.72:0/2627253376 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3639806169 0 0) 0x7faf000afd50 con 0x7faf0013ef30 2024-03-29T11:44:36.217 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.209+0000 7faefeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:44:36.217 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.209+0000 7faefeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:36.218 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.209+0000 7faefeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:36.218 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.209+0000 7faefeffd640 10 monclient(hunting): my global_id is 4586 2024-03-29T11:44:36.218 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.209+0000 7faefeffd640 1 -- 172.21.15.72:0/2627253376 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf000b0c40 con 0x7faf0013ef30 2024-03-29T11:44:36.218 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.209+0000 7faefeffd640 1 -- 172.21.15.72:0/2627253376 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3138848304 0 0) 0x7faf000aed10 con 0x7faf000b0860 2024-03-29T11:44:36.218 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.209+0000 7faefeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:44:36.219 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.209+0000 7faefeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:36.219 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.209+0000 7faefeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:36.219 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.209+0000 7faefeffd640 10 monclient(hunting): my global_id is 4513 2024-03-29T11:44:36.219 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.209+0000 7faefeffd640 1 -- 172.21.15.72:0/2627253376 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf000afd50 con 0x7faf000b0860 2024-03-29T11:44:36.219 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.209+0000 7faefeffd640 1 -- 172.21.15.72:0/2627253376 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3336591068 0 0) 0x7faf000afd50 con 0x7faf000b0860 2024-03-29T11:44:36.219 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.209+0000 7faefeffd640 1 -- 172.21.15.72:0/2627253376 >> v1:172.21.15.72:6790/0 conn(0x7faf0013ef30 legacy=0x7faf000af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:36.219 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.209+0000 7faefeffd640 1 -- 172.21.15.72:0/2627253376 >> v1:172.21.15.72:6789/0 conn(0x7faf000af8e0 legacy=0x7faf00149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:36.219 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.209+0000 7faefeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:44:36.219 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.209+0000 7faefeffd640 1 monclient: found mon.noname-c 2024-03-29T11:44:36.220 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.209+0000 7faefeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:36.220 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.209+0000 7faefeffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.89:6789/0 2024-03-29T11:44:36.220 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.209+0000 7faefeffd640 1 -- 172.21.15.72:0/2627253376 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf000ab1b0 con 0x7faf000b0860 2024-03-29T11:44:36.220 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.209+0000 7faefeffd640 10 monclient: _finish_auth 0 2024-03-29T11:44:36.220 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.209+0000 7faefeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:44:36.220 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.209+0000 7faefeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:36.220 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.209+0000 7faf06979640 5 monclient: authenticate success, global_id 4513 2024-03-29T11:44:36.220 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.209+0000 7faf06979640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:36.220 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.209+0000 7faefeffd640 1 -- 172.21.15.72:0/2627253376 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7faef00015b0 con 0x7faf000b0860 2024-03-29T11:44:36.221 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.209+0000 7faefeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:36.221 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.209+0000 7faefeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:44:36.221 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.209+0000 7faefeffd640 10 monclient: dump: 2024-03-29T11:44:36.221 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:44:36.221 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:44:36.221 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:36.221 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:36.221 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:36.221 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:36.222 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:44:36.222 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:44:36.222 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:44:36.222 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:36.222 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.209+0000 7faf06979640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:36.222 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.209+0000 7faefeffd640 1 -- 172.21.15.72:0/2627253376 <== mon.1 v1:172.21.15.89:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faef00017d0 con 0x7faf000b0860 2024-03-29T11:44:36.222 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.209+0000 7faefeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:36.222 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.209+0000 7faf06979640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:36.222 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.209+0000 7faf06979640 4 set_mon_vals no callback set 2024-03-29T11:44:36.222 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.210+0000 7faf06979640 10 monclient: shutdown 2024-03-29T11:44:36.222 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.210+0000 7faf06979640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:36.223 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.210+0000 7faf06979640 1 -- 172.21.15.72:0/2627253376 >> v1:172.21.15.89:6789/0 conn(0x7faf000b0860 legacy=0x7faf0013e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:36.223 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.211+0000 7faf06979640 1 -- 172.21.15.72:0/2627253376 shutdown_connections 2024-03-29T11:44:36.223 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.212+0000 7faf06979640 1 -- 172.21.15.72:0/2627253376 >> 172.21.15.72:0/2627253376 conn(0x7faf0006b660 msgr2=0x7faf0006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:36.224 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.217+0000 7faf06979640 1 -- 172.21.15.72:0/2627253376 shutdown_connections 2024-03-29T11:44:36.224 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.217+0000 7faf06979640 1 -- 172.21.15.72:0/2627253376 wait complete. 2024-03-29T11:44:36.224 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.218+0000 7faf06979640 10 monclient: build_initial_monmap 2024-03-29T11:44:36.224 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.218+0000 7faf06979640 10 monclient: monmap: 2024-03-29T11:44:36.224 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:44:36.225 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:36.225 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:44:36.225 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:44:36.225 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:36.225 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:36.225 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:44:36.225 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:44:36.225 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:44:36.225 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:36.225 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.218+0000 7faf06979640 1 librados: starting msgr at 2024-03-29T11:44:36.225 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.218+0000 7faf06979640 1 librados: starting objecter 2024-03-29T11:44:36.225 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.218+0000 7faf06979640 1 Processor -- start 2024-03-29T11:44:36.225 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.219+0000 7faf06979640 1 -- start start 2024-03-29T11:44:36.225 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.219+0000 7faf06979640 1 librados: setting wanted keys 2024-03-29T11:44:36.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.219+0000 7faf06979640 1 librados: calling monclient init 2024-03-29T11:44:36.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.219+0000 7faf06979640 10 monclient: init 2024-03-29T11:44:36.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.220+0000 7faf06979640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:36.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.220+0000 7faf06979640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:44:36.226 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.220+0000 7faf06979640 10 monclient(hunting): picked mon.noname-b con 0x7faf000af8e0 addr v1:172.21.15.89:6789/0 2024-03-29T11:44:36.227 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.223+0000 7faf06979640 10 monclient(hunting): picked mon.noname-c con 0x7faf000b0860 addr v1:172.21.15.72:6790/0 2024-03-29T11:44:36.227 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.223+0000 7faeff7fe640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7faf000b0860 0x7faf00071240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:40393/0 (socket says 172.21.15.72:40393) 2024-03-29T11:44:36.227 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.223+0000 7faeff7fe640 1 -- 172.21.15.72:0/1273897846 learned_addr learned my addr 172.21.15.72:0/1273897846 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:44:36.237 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.223+0000 7faf06979640 10 monclient(hunting): picked mon.noname-a con 0x7faf0013ef30 addr v1:172.21.15.72:6789/0 2024-03-29T11:44:36.237 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.227+0000 7faf06979640 1 -- 172.21.15.72:0/1273897846 --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf0006f540 con 0x7faf0013ef30 2024-03-29T11:44:36.237 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.227+0000 7faf06979640 1 -- 172.21.15.72:0/1273897846 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf001d7610 con 0x7faf000af8e0 2024-03-29T11:44:36.237 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.227+0000 7faf06979640 1 -- 172.21.15.72:0/1273897846 --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf00071950 con 0x7faf000b0860 2024-03-29T11:44:36.237 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.227+0000 7faf06979640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:36.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.228+0000 7faf06979640 10 monclient(hunting): authenticate will time out at 1435.516019s 2024-03-29T11:44:36.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.228+0000 7faefcff9640 10 client.?.objecter ms_handle_connect 0x7faf000b0860 2024-03-29T11:44:36.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.228+0000 7faefcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:36.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.228+0000 7faefcff9640 1 -- 172.21.15.72:0/1273897846 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 737778904 0 0) 0x7faf00071950 con 0x7faf000b0860 2024-03-29T11:44:36.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.228+0000 7faefcff9640 10 monclient(hunting): _negotiate 2024-03-29T11:44:36.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.228+0000 7faefcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:36.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.228+0000 7faefcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:36.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.228+0000 7faefcff9640 10 monclient(hunting): my global_id is 4589 2024-03-29T11:44:36.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.228+0000 7faefcff9640 1 -- 172.21.15.72:0/1273897846 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faeec004eb0 con 0x7faf000b0860 2024-03-29T11:44:36.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.228+0000 7faefcff9640 10 client.?.objecter ms_handle_connect 0x7faf0013ef30 2024-03-29T11:44:36.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.229+0000 7faefcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:36.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.229+0000 7faefcff9640 1 -- 172.21.15.72:0/1273897846 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 554273845 0 0) 0x7faeec004eb0 con 0x7faf000b0860 2024-03-29T11:44:36.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.229+0000 7faefcff9640 1 -- 172.21.15.72:0/1273897846 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf00071950 con 0x7faf000b0860 2024-03-29T11:44:36.238 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.229+0000 7faefcff9640 1 -- 172.21.15.72:0/1273897846 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4086645889 0 0) 0x7faf0006f540 con 0x7faf0013ef30 2024-03-29T11:44:36.239 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.229+0000 7faefcff9640 10 monclient(hunting): _negotiate 2024-03-29T11:44:36.239 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.229+0000 7faefcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:36.239 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.229+0000 7faefcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:36.239 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.229+0000 7faefcff9640 10 monclient(hunting): my global_id is 4560 2024-03-29T11:44:36.239 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.229+0000 7faefcff9640 1 -- 172.21.15.72:0/1273897846 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faeec004eb0 con 0x7faf0013ef30 2024-03-29T11:44:36.239 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.229+0000 7faefcff9640 1 -- 172.21.15.72:0/1273897846 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7faef40018d0 con 0x7faf000b0860 2024-03-29T11:44:36.239 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.230+0000 7faefcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:36.239 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.230+0000 7faefcff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:44:36.239 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.230+0000 7faefcff9640 10 monclient(hunting): dump: 2024-03-29T11:44:36.239 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:44:36.239 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:44:36.239 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:36.239 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:36.239 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:36.239 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:36.240 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:44:36.240 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:44:36.240 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:44:36.240 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:36.240 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.230+0000 7faefcff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:36.240 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.230+0000 7faefcff9640 1 -- 172.21.15.72:0/1273897846 <== mon.2 v1:172.21.15.72:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2755619105 0 0) 0x7faf00071950 con 0x7faf000b0860 2024-03-29T11:44:36.240 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.230+0000 7faefcff9640 1 -- 172.21.15.72:0/1273897846 >> v1:172.21.15.89:6789/0 conn(0x7faf000af8e0 legacy=0x7faf0006de90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:44:36.240 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.230+0000 7faefcff9640 1 -- 172.21.15.72:0/1273897846 >> v1:172.21.15.72:6789/0 conn(0x7faf0013ef30 legacy=0x7faf001d7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:36.240 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.230+0000 7faefcff9640 10 monclient: _finish_hunting 0 2024-03-29T11:44:36.240 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.231+0000 7faefcff9640 1 monclient: found mon.c 2024-03-29T11:44:36.240 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.231+0000 7faefcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:36.240 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.231+0000 7faefcff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:44:36.240 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.231+0000 7faefcff9640 1 -- 172.21.15.72:0/1273897846 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf00071ae0 con 0x7faf000b0860 2024-03-29T11:44:36.241 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.231+0000 7faefcff9640 10 monclient: _finish_auth 0 2024-03-29T11:44:36.241 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.231+0000 7faefcff9640 10 monclient: _check_auth_tickets 2024-03-29T11:44:36.241 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.231+0000 7faefcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:36.241 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.231+0000 7faf06979640 5 monclient: authenticate success, global_id 4589 2024-03-29T11:44:36.241 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.231+0000 7faf06979640 10 monclient: _renew_subs 2024-03-29T11:44:36.241 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.231+0000 7faf06979640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:44:36.241 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.231+0000 7faf06979640 1 -- 172.21.15.72:0/1273897846 --> v1:172.21.15.72:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf001d8240 con 0x7faf000b0860 2024-03-29T11:44:36.241 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.231+0000 7faf06979640 10 client.4589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:36.241 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.231+0000 7faf06979640 10 monclient: _renew_subs 2024-03-29T11:44:36.241 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.231+0000 7faf06979640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:44:36.241 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.231+0000 7faf06979640 1 -- 172.21.15.72:0/1273897846 --> v1:172.21.15.72:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faf00135be0 con 0x7faf000b0860 2024-03-29T11:44:36.241 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.231+0000 7faf06979640 1 librados: init done 2024-03-29T11:44:36.241 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.231+0000 7faefcff9640 1 -- 172.21.15.72:0/1273897846 <== mon.2 v1:172.21.15.72:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faef4001aa0 con 0x7faf000b0860 2024-03-29T11:44:36.241 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.231+0000 7faefcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:36.241 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.231+0000 7faefcff9640 1 -- 172.21.15.72:0/1273897846 <== mon.2 v1:172.21.15.72:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7faef4001cd0 con 0x7faf000b0860 2024-03-29T11:44:36.241 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.231+0000 7faefcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:36.242 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.231+0000 7faefcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:36.242 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.231+0000 7faefcff9640 10 monclient: dump: 2024-03-29T11:44:36.242 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:44:36.242 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:44:36.242 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:36.242 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:36.242 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:36.242 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:36.242 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:44:36.242 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:44:36.242 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:44:36.242 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:36.242 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.231+0000 7faefe7fc640 4 set_mon_vals no callback set 2024-03-29T11:44:36.243 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.237+0000 7faefcff9640 1 -- 172.21.15.72:0/1273897846 <== mon.2 v1:172.21.15.72:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7faef40038c0 con 0x7faf000b0860 2024-03-29T11:44:36.244 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.238+0000 7faefcff9640 1 -- 172.21.15.72:0/1273897846 <== mon.2 v1:172.21.15.72:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7faef40847f0 con 0x7faf000b0860 2024-03-29T11:44:36.244 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.238+0000 7faefcff9640 10 client.4589.objecter ms_dispatch 0x7faf00000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:36.244 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.238+0000 7faefcff9640 3 client.4589.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:36.244 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.238+0000 7faefcff9640 3 client.4589.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:36.244 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.238+0000 7faefcff9640 20 client.4589.objecter dump_active .. 0 homeless 2024-03-29T11:44:36.244 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.238+0000 7faefffff640 10 monclient: get_auth_request con 0x7faeec06dc30 auth_method 0 2024-03-29T11:44:36.244 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.239+0000 7faefcff9640 10 client.4589.objecter ms_handle_connect 0x7faeec06dc30 2024-03-29T11:44:36.245 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.241+0000 7faf06979640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:36.245 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.241+0000 7faf06979640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:36.245 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.241+0000 7faf06979640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:44:36.245 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.241+0000 7faf06979640 1 -- 172.21.15.72:0/1273897846 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faee0005080 con 0x7faf000b0860 2024-03-29T11:44:36.257 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.253+0000 7faefcff9640 1 -- 172.21.15.72:0/1273897846 <== mon.2 v1:172.21.15.72:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faef40500d0 con 0x7faf000b0860 2024-03-29T11:44:36.257 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.253+0000 7faefcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:36.257 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.253+0000 7faefcff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:36.634 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.629+0000 7faf06979640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:36.635 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.629+0000 7faf06979640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:36.635 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.629+0000 7faf06979640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:44:36.635 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.629+0000 7faf06979640 1 -- 172.21.15.72:0/1273897846 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faee0005330 con 0x7faf000b0860 2024-03-29T11:44:36.635 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.630+0000 7faefcff9640 1 -- 172.21.15.72:0/1273897846 <== mon.2 v1:172.21.15.72:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7faef4052000 con 0x7faf000b0860 2024-03-29T11:44:36.635 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.630+0000 7faefcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:36.635 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.630+0000 7faefcff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:36.635 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:44:36.635 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:36.638 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.634+0000 7faf06979640 10 librados: watch_flush enter 2024-03-29T11:44:36.638 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.634+0000 7faf06979640 10 librados: watch_flush exit 2024-03-29T11:44:36.638 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.634+0000 7faf06979640 20 client.4589.objecter shutdown clearing up homeless session... 2024-03-29T11:44:36.638 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.634+0000 7faf06979640 10 client.4589.objecter successfully canceled tick 2024-03-29T11:44:36.638 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.634+0000 7faf06979640 1 -- 172.21.15.72:0/1273897846 >> v1:172.21.15.72:6816/1578866102 conn(0x7faeec06dc30 legacy=0x7faeec0700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:36.639 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.634+0000 7faf06979640 10 monclient: shutdown 2024-03-29T11:44:36.639 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.634+0000 7faf06979640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:36.639 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.634+0000 7faf06979640 1 -- 172.21.15.72:0/1273897846 >> v1:172.21.15.72:6790/0 conn(0x7faf000b0860 legacy=0x7faf00071240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:36.639 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.634+0000 7faf06979640 1 -- 172.21.15.72:0/1273897846 shutdown_connections 2024-03-29T11:44:36.639 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.634+0000 7faf06979640 1 -- 172.21.15.72:0/1273897846 >> 172.21.15.72:0/1273897846 conn(0x7faf0006b660 msgr2=0x7faf000ac070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:36.639 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.634+0000 7faf06979640 1 -- 172.21.15.72:0/1273897846 shutdown_connections 2024-03-29T11:44:36.639 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.634+0000 7faf06979640 1 -- 172.21.15.72:0/1273897846 wait complete. 2024-03-29T11:44:36.639 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:36.634+0000 7faf06979640 1 librados: shutdown 2024-03-29T11:44:36.655 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:43.462 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:43.616 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.611+0000 7fc09439c640 10 monclient: get_monmap_and_config 2024-03-29T11:44:43.616 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.611+0000 7fc09439c640 10 monclient: build_initial_monmap 2024-03-29T11:44:43.616 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.611+0000 7fc09439c640 10 monclient: monmap: 2024-03-29T11:44:43.616 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:44:43.616 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:44:43.616 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:44:43.612888+0000 2024-03-29T11:44:43.616 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:44:43.612888+0000 2024-03-29T11:44:43.616 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:43.617 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:43.617 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:44:43.617 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:44:43.617 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:44:43.617 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:43.619 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.614+0000 7fc09439c640 1 Processor -- start 2024-03-29T11:44:43.619 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.614+0000 7fc09439c640 1 -- start start 2024-03-29T11:44:43.619 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.614+0000 7fc09439c640 10 monclient: init 2024-03-29T11:44:43.620 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.615+0000 7fc09439c640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:43.620 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.615+0000 7fc09439c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:44:43.620 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.615+0000 7fc09439c640 10 monclient(hunting): picked mon.noname-b con 0x7fc08c0b0860 addr v1:172.21.15.72:6790/0 2024-03-29T11:44:43.620 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.615+0000 7fc09439c640 10 monclient(hunting): picked mon.noname-c con 0x7fc08c13ef30 addr v1:172.21.15.89:6789/0 2024-03-29T11:44:43.620 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.615+0000 7fc09439c640 10 monclient(hunting): picked mon.noname-a con 0x7fc08c0af8e0 addr v1:172.21.15.72:6789/0 2024-03-29T11:44:43.620 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.615+0000 7fc09439c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc08c0b0c40 con 0x7fc08c0af8e0 2024-03-29T11:44:43.620 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.615+0000 7fc09439c640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc08c0aed10 con 0x7fc08c13ef30 2024-03-29T11:44:43.620 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.615+0000 7fc09439c640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc08c0afd50 con 0x7fc08c0b0860 2024-03-29T11:44:43.620 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.615+0000 7fc09439c640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:43.620 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.615+0000 7fc09439c640 10 monclient(hunting): authenticate will time out at 1442.903866s 2024-03-29T11:44:43.624 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.619+0000 7fc091937640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fc08c13ef30 0x7fc08c0af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:56655/0 (socket says 172.21.15.72:56655) 2024-03-29T11:44:43.624 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.619+0000 7fc091937640 1 -- 172.21.15.72:0/2966538657 learned_addr learned my addr 172.21.15.72:0/2966538657 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:44:43.627 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.623+0000 7fc091136640 1 -- 172.21.15.72:0/2966538657 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3966657812 0 0) 0x7fc08c0aed10 con 0x7fc08c13ef30 2024-03-29T11:44:43.627 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.623+0000 7fc091136640 10 monclient(hunting): _negotiate 2024-03-29T11:44:43.628 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.623+0000 7fc091136640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:43.628 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.623+0000 7fc091136640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:43.628 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.623+0000 7fc091136640 10 monclient(hunting): my global_id is 4516 2024-03-29T11:44:43.628 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.623+0000 7fc091136640 1 -- 172.21.15.72:0/2966538657 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc078003520 con 0x7fc08c13ef30 2024-03-29T11:44:43.628 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.623+0000 7fc091136640 1 -- 172.21.15.72:0/2966538657 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 560233852 0 0) 0x7fc08c0b0c40 con 0x7fc08c0af8e0 2024-03-29T11:44:43.628 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.623+0000 7fc091136640 10 monclient(hunting): _negotiate 2024-03-29T11:44:43.628 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.623+0000 7fc091136640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:43.628 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.623+0000 7fc091136640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:43.628 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.623+0000 7fc091136640 10 monclient(hunting): my global_id is 4563 2024-03-29T11:44:43.628 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.623+0000 7fc091136640 1 -- 172.21.15.72:0/2966538657 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc08c0aed10 con 0x7fc08c0af8e0 2024-03-29T11:44:43.628 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.623+0000 7fc091136640 1 -- 172.21.15.72:0/2966538657 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1532570578 0 0) 0x7fc08c0afd50 con 0x7fc08c0b0860 2024-03-29T11:44:43.628 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.623+0000 7fc091136640 10 monclient(hunting): _negotiate 2024-03-29T11:44:43.628 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.623+0000 7fc091136640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:43.629 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.623+0000 7fc091136640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:43.629 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.623+0000 7fc091136640 10 monclient(hunting): my global_id is 4592 2024-03-29T11:44:43.629 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.623+0000 7fc091136640 1 -- 172.21.15.72:0/2966538657 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc08c0b0c40 con 0x7fc08c0b0860 2024-03-29T11:44:43.629 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.623+0000 7fc091136640 1 -- 172.21.15.72:0/2966538657 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 317347921 0 0) 0x7fc08c0aed10 con 0x7fc08c0af8e0 2024-03-29T11:44:43.629 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.623+0000 7fc091136640 1 -- 172.21.15.72:0/2966538657 >> v1:172.21.15.72:6790/0 conn(0x7fc08c0b0860 legacy=0x7fc08c13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:43.629 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.623+0000 7fc091136640 1 -- 172.21.15.72:0/2966538657 >> v1:172.21.15.89:6789/0 conn(0x7fc08c13ef30 legacy=0x7fc08c0af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:43.629 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.623+0000 7fc091136640 10 monclient: _finish_hunting 0 2024-03-29T11:44:43.629 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.623+0000 7fc091136640 1 monclient: found mon.noname-a 2024-03-29T11:44:43.629 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.623+0000 7fc091136640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:43.629 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.623+0000 7fc091136640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.72:6789/0 2024-03-29T11:44:43.629 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.623+0000 7fc091136640 1 -- 172.21.15.72:0/2966538657 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc08c0ab1b0 con 0x7fc08c0af8e0 2024-03-29T11:44:43.629 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.623+0000 7fc091136640 10 monclient: _finish_auth 0 2024-03-29T11:44:43.629 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.623+0000 7fc091136640 10 monclient: _check_auth_tickets 2024-03-29T11:44:43.629 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.623+0000 7fc091136640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:43.629 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.623+0000 7fc091136640 1 -- 172.21.15.72:0/2966538657 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fc07c001670 con 0x7fc08c0af8e0 2024-03-29T11:44:43.629 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.623+0000 7fc09439c640 5 monclient: authenticate success, global_id 4563 2024-03-29T11:44:43.630 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.623+0000 7fc09439c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:43.630 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.623+0000 7fc091136640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:43.630 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.623+0000 7fc091136640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:44:43.630 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.623+0000 7fc091136640 10 monclient: dump: 2024-03-29T11:44:43.630 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:44:43.630 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:44:43.630 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:43.630 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:43.630 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:43.630 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:43.630 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:44:43.630 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:44:43.630 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:44:43.630 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:43.630 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.623+0000 7fc091136640 1 -- 172.21.15.72:0/2966538657 <== mon.0 v1:172.21.15.72:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc07c0017d0 con 0x7fc08c0af8e0 2024-03-29T11:44:43.631 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.623+0000 7fc091136640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:43.631 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.623+0000 7fc091136640 1 -- 172.21.15.72:0/2966538657 <== mon.0 v1:172.21.15.72:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fc07c001be0 con 0x7fc08c0af8e0 2024-03-29T11:44:43.631 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.623+0000 7fc091136640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:43.631 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.623+0000 7fc091136640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:43.631 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.623+0000 7fc091136640 10 monclient: dump: 2024-03-29T11:44:43.631 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:44:43.631 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:44:43.631 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:43.631 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:43.631 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:43.631 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:43.632 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:44:43.632 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:44:43.632 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:44:43.632 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:43.632 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.624+0000 7fc09439c640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:43.632 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.624+0000 7fc09439c640 4 set_mon_vals no callback set 2024-03-29T11:44:43.634 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.624+0000 7fc09439c640 10 monclient: shutdown 2024-03-29T11:44:43.634 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.624+0000 7fc09439c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:43.634 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.624+0000 7fc09439c640 1 -- 172.21.15.72:0/2966538657 >> v1:172.21.15.72:6789/0 conn(0x7fc08c0af8e0 legacy=0x7fc08c149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:43.634 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.625+0000 7fc09439c640 1 -- 172.21.15.72:0/2966538657 shutdown_connections 2024-03-29T11:44:43.634 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.625+0000 7fc09439c640 1 -- 172.21.15.72:0/2966538657 >> 172.21.15.72:0/2966538657 conn(0x7fc08c06b660 msgr2=0x7fc08c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:43.634 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.625+0000 7fc09439c640 1 -- 172.21.15.72:0/2966538657 shutdown_connections 2024-03-29T11:44:43.634 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.626+0000 7fc09439c640 1 -- 172.21.15.72:0/2966538657 wait complete. 2024-03-29T11:44:43.634 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.627+0000 7fc09439c640 10 monclient: build_initial_monmap 2024-03-29T11:44:43.634 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.627+0000 7fc09439c640 10 monclient: monmap: 2024-03-29T11:44:43.634 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:44:43.634 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:43.634 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:44:43.635 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:44:43.635 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:43.635 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:43.635 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:44:43.635 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:44:43.635 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:44:43.635 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:43.635 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.628+0000 7fc09439c640 1 librados: starting msgr at 2024-03-29T11:44:43.635 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.628+0000 7fc09439c640 1 librados: starting objecter 2024-03-29T11:44:43.635 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.628+0000 7fc09439c640 1 Processor -- start 2024-03-29T11:44:43.635 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.628+0000 7fc09439c640 1 -- start start 2024-03-29T11:44:43.636 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.628+0000 7fc09439c640 1 librados: setting wanted keys 2024-03-29T11:44:43.636 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.628+0000 7fc09439c640 1 librados: calling monclient init 2024-03-29T11:44:43.636 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.628+0000 7fc09439c640 10 monclient: init 2024-03-29T11:44:43.636 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.629+0000 7fc09439c640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:43.636 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.629+0000 7fc09439c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:44:43.636 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.629+0000 7fc09439c640 10 monclient(hunting): picked mon.noname-b con 0x7fc08c0af8e0 addr v1:172.21.15.89:6789/0 2024-03-29T11:44:43.636 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.629+0000 7fc09439c640 10 monclient(hunting): picked mon.noname-a con 0x7fc08c0b0860 addr v1:172.21.15.72:6789/0 2024-03-29T11:44:43.636 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.629+0000 7fc09439c640 10 monclient(hunting): picked mon.noname-c con 0x7fc08c13ef30 addr v1:172.21.15.72:6790/0 2024-03-29T11:44:43.636 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.629+0000 7fc09439c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc08c0b3560 con 0x7fc08c0b0860 2024-03-29T11:44:43.636 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.629+0000 7fc09439c640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc08c0b3b10 con 0x7fc08c0af8e0 2024-03-29T11:44:43.636 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.629+0000 7fc09439c640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc08c0b3ca0 con 0x7fc08c13ef30 2024-03-29T11:44:43.636 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.629+0000 7fc09439c640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:43.636 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.629+0000 7fc09439c640 10 monclient(hunting): authenticate will time out at 1442.917619s 2024-03-29T11:44:43.639 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.634+0000 7fc091937640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fc08c0b0860 0x7fc08c1d3120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37117/0 (socket says 172.21.15.72:37117) 2024-03-29T11:44:43.639 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.634+0000 7fc091937640 1 -- 172.21.15.72:0/2561677902 learned_addr learned my addr 172.21.15.72:0/2561677902 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:44:43.639 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.635+0000 7fc082ffd640 10 client.?.objecter ms_handle_connect 0x7fc08c0b0860 2024-03-29T11:44:43.639 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.635+0000 7fc082ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:43.639 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.635+0000 7fc082ffd640 10 client.?.objecter ms_handle_connect 0x7fc08c0af8e0 2024-03-29T11:44:43.640 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.635+0000 7fc082ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:43.640 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.635+0000 7fc082ffd640 1 -- 172.21.15.72:0/2561677902 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3348684311 0 0) 0x7fc08c0b3560 con 0x7fc08c0b0860 2024-03-29T11:44:43.640 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.635+0000 7fc082ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:44:43.640 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.635+0000 7fc082ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:43.640 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.635+0000 7fc082ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:43.640 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.635+0000 7fc082ffd640 10 monclient(hunting): my global_id is 4566 2024-03-29T11:44:43.640 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.635+0000 7fc082ffd640 1 -- 172.21.15.72:0/2561677902 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc070004eb0 con 0x7fc08c0b0860 2024-03-29T11:44:43.640 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.636+0000 7fc082ffd640 1 -- 172.21.15.72:0/2561677902 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1480412129 0 0) 0x7fc08c0b3b10 con 0x7fc08c0af8e0 2024-03-29T11:44:43.640 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.636+0000 7fc082ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:44:43.640 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.636+0000 7fc082ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:43.640 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.636+0000 7fc082ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:43.640 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.636+0000 7fc082ffd640 10 monclient(hunting): my global_id is 4519 2024-03-29T11:44:43.640 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.636+0000 7fc082ffd640 1 -- 172.21.15.72:0/2561677902 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc08c0b3560 con 0x7fc08c0af8e0 2024-03-29T11:44:43.641 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.637+0000 7fc082ffd640 1 -- 172.21.15.72:0/2561677902 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3589193908 0 0) 0x7fc070004eb0 con 0x7fc08c0b0860 2024-03-29T11:44:43.641 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.637+0000 7fc082ffd640 1 -- 172.21.15.72:0/2561677902 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc08c0b3b10 con 0x7fc08c0b0860 2024-03-29T11:44:43.641 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.637+0000 7fc082ffd640 1 -- 172.21.15.72:0/2561677902 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3330604873 0 0) 0x7fc08c0b3560 con 0x7fc08c0af8e0 2024-03-29T11:44:43.641 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.637+0000 7fc082ffd640 1 -- 172.21.15.72:0/2561677902 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc070004eb0 con 0x7fc08c0af8e0 2024-03-29T11:44:43.641 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.637+0000 7fc082ffd640 1 -- 172.21.15.72:0/2561677902 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fc084006210 con 0x7fc08c0b0860 2024-03-29T11:44:43.642 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.637+0000 7fc082ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:43.642 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.637+0000 7fc082ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:44:43.642 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.637+0000 7fc082ffd640 10 monclient(hunting): dump: 2024-03-29T11:44:43.642 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:44:43.642 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:44:43.642 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:43.642 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:43.642 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:43.642 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:43.642 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:44:43.642 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:44:43.642 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:44:43.642 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:43.642 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.637+0000 7fc082ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:43.643 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.637+0000 7fc082ffd640 1 -- 172.21.15.72:0/2561677902 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fc088001830 con 0x7fc08c0af8e0 2024-03-29T11:44:43.643 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.637+0000 7fc082ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:43.643 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.637+0000 7fc082ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:44:43.643 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.637+0000 7fc082ffd640 10 monclient(hunting): dump: 2024-03-29T11:44:43.643 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:44:43.643 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:44:43.643 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:43.643 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:43.643 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:43.643 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:43.643 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:44:43.643 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:44:43.643 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:44:43.643 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:43.645 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.638+0000 7fc082ffd640 1 -- 172.21.15.72:0/2561677902 <== mon.0 v1:172.21.15.72:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1096065356 0 0) 0x7fc08c0b3b10 con 0x7fc08c0b0860 2024-03-29T11:44:43.645 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.638+0000 7fc082ffd640 1 -- 172.21.15.72:0/2561677902 >> v1:172.21.15.72:6790/0 conn(0x7fc08c13ef30 legacy=0x7fc08c1d3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:43.645 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.638+0000 7fc082ffd640 1 -- 172.21.15.72:0/2561677902 >> v1:172.21.15.89:6789/0 conn(0x7fc08c0af8e0 legacy=0x7fc08c1346a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:43.645 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.638+0000 7fc082ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:44:43.645 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.638+0000 7fc082ffd640 1 monclient: found mon.a 2024-03-29T11:44:43.645 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.638+0000 7fc082ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:43.645 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.638+0000 7fc082ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:44:43.645 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.638+0000 7fc082ffd640 1 -- 172.21.15.72:0/2561677902 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc08c0b3e30 con 0x7fc08c0b0860 2024-03-29T11:44:43.645 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.638+0000 7fc082ffd640 10 monclient: _finish_auth 0 2024-03-29T11:44:43.645 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.638+0000 7fc082ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:44:43.646 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.638+0000 7fc082ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:43.646 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.638+0000 7fc09439c640 5 monclient: authenticate success, global_id 4566 2024-03-29T11:44:43.646 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.638+0000 7fc09439c640 10 monclient: _renew_subs 2024-03-29T11:44:43.646 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.638+0000 7fc09439c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:44:43.646 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.638+0000 7fc09439c640 1 -- 172.21.15.72:0/2561677902 --> v1:172.21.15.72:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc08c1c8450 con 0x7fc08c0b0860 2024-03-29T11:44:43.646 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.638+0000 7fc09439c640 10 client.4566.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:43.646 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.638+0000 7fc09439c640 10 monclient: _renew_subs 2024-03-29T11:44:43.646 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.638+0000 7fc09439c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:44:43.646 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.638+0000 7fc09439c640 1 -- 172.21.15.72:0/2561677902 --> v1:172.21.15.72:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc08c0b30b0 con 0x7fc08c0b0860 2024-03-29T11:44:43.646 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.638+0000 7fc09439c640 1 librados: init done 2024-03-29T11:44:43.646 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.639+0000 7fc082ffd640 1 -- 172.21.15.72:0/2561677902 <== mon.0 v1:172.21.15.72:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc084001820 con 0x7fc08c0b0860 2024-03-29T11:44:43.646 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.639+0000 7fc082ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:43.646 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.639+0000 7fc082ffd640 1 -- 172.21.15.72:0/2561677902 <== mon.0 v1:172.21.15.72:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fc084001c30 con 0x7fc08c0b0860 2024-03-29T11:44:43.646 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.639+0000 7fc082ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:43.647 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.639+0000 7fc082ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:43.647 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.639+0000 7fc082ffd640 10 monclient: dump: 2024-03-29T11:44:43.647 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:44:43.647 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:44:43.647 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:43.647 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:43.647 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:43.647 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:43.647 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:44:43.647 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:44:43.647 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:44:43.647 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:43.647 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.639+0000 7fc09439c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:43.647 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.639+0000 7fc09439c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:43.647 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.639+0000 7fc09439c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:44:43.647 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.639+0000 7fc09439c640 1 -- 172.21.15.72:0/2561677902 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc08c0b33b0 con 0x7fc08c0b0860 2024-03-29T11:44:43.649 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.640+0000 7fc090935640 4 set_mon_vals no callback set 2024-03-29T11:44:43.649 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.642+0000 7fc082ffd640 1 -- 172.21.15.72:0/2561677902 <== mon.0 v1:172.21.15.72:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7fc084001d90 con 0x7fc08c0b0860 2024-03-29T11:44:43.649 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.643+0000 7fc092138640 10 monclient: get_auth_request con 0x7fc07006ddc0 auth_method 0 2024-03-29T11:44:43.649 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.643+0000 7fc082ffd640 1 -- 172.21.15.72:0/2561677902 <== mon.0 v1:172.21.15.72:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7fc084084d70 con 0x7fc08c0b0860 2024-03-29T11:44:43.649 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.643+0000 7fc082ffd640 10 client.4566.objecter ms_dispatch 0x7fc08c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:43.649 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.643+0000 7fc082ffd640 3 client.4566.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:43.649 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.643+0000 7fc082ffd640 3 client.4566.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:43.649 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.643+0000 7fc082ffd640 20 client.4566.objecter dump_active .. 0 homeless 2024-03-29T11:44:43.649 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.644+0000 7fc082ffd640 10 client.4566.objecter ms_handle_connect 0x7fc07006ddc0 2024-03-29T11:44:43.654 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.650+0000 7fc082ffd640 1 -- 172.21.15.72:0/2561677902 <== mon.0 v1:172.21.15.72:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc084050650 con 0x7fc08c0b0860 2024-03-29T11:44:43.654 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.650+0000 7fc082ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:43.654 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.650+0000 7fc082ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:43.992 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.988+0000 7fc09439c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:43.992 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.988+0000 7fc09439c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:43.992 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.988+0000 7fc09439c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:44:43.992 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.988+0000 7fc09439c640 1 -- 172.21.15.72:0/2561677902 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc08c1c87f0 con 0x7fc08c0b0860 2024-03-29T11:44:43.994 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.991+0000 7fc082ffd640 1 -- 172.21.15.72:0/2561677902 <== mon.0 v1:172.21.15.72:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7fc0840020c0 con 0x7fc08c0b0860 2024-03-29T11:44:43.995 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.991+0000 7fc082ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:43.995 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.991+0000 7fc082ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:43.997 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:44:43.997 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:44.000 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.996+0000 7fc09439c640 10 librados: watch_flush enter 2024-03-29T11:44:44.001 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.996+0000 7fc09439c640 10 librados: watch_flush exit 2024-03-29T11:44:44.001 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.996+0000 7fc09439c640 20 client.4566.objecter shutdown clearing up homeless session... 2024-03-29T11:44:44.001 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.996+0000 7fc09439c640 10 client.4566.objecter successfully canceled tick 2024-03-29T11:44:44.001 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.996+0000 7fc09439c640 1 -- 172.21.15.72:0/2561677902 >> v1:172.21.15.72:6816/1578866102 conn(0x7fc07006ddc0 legacy=0x7fc070070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:44.001 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.996+0000 7fc09439c640 10 monclient: shutdown 2024-03-29T11:44:44.001 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.996+0000 7fc09439c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:44.001 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.996+0000 7fc09439c640 1 -- 172.21.15.72:0/2561677902 >> v1:172.21.15.72:6789/0 conn(0x7fc08c0b0860 legacy=0x7fc08c1d3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:44.001 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.996+0000 7fc09439c640 1 -- 172.21.15.72:0/2561677902 shutdown_connections 2024-03-29T11:44:44.001 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.996+0000 7fc09439c640 1 -- 172.21.15.72:0/2561677902 >> 172.21.15.72:0/2561677902 conn(0x7fc08c06b660 msgr2=0x7fc08c13e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:44.003 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.997+0000 7fc09439c640 1 -- 172.21.15.72:0/2561677902 shutdown_connections 2024-03-29T11:44:44.004 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.997+0000 7fc09439c640 1 -- 172.21.15.72:0/2561677902 wait complete. 2024-03-29T11:44:44.004 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:43.997+0000 7fc09439c640 1 librados: shutdown 2024-03-29T11:44:44.022 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:50.828 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:50.959 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.955+0000 7f8e39d3b640 10 monclient: get_monmap_and_config 2024-03-29T11:44:50.960 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.955+0000 7f8e39d3b640 10 monclient: build_initial_monmap 2024-03-29T11:44:50.960 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.955+0000 7f8e39d3b640 10 monclient: monmap: 2024-03-29T11:44:50.960 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:44:50.960 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:44:50.960 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:44:50.956663+0000 2024-03-29T11:44:50.960 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:44:50.956663+0000 2024-03-29T11:44:50.960 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:50.960 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:50.960 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:44:50.960 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:44:50.960 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:44:50.960 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:50.961 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.956+0000 7f8e39d3b640 1 Processor -- start 2024-03-29T11:44:50.961 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.956+0000 7f8e39d3b640 1 -- start start 2024-03-29T11:44:50.961 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.956+0000 7f8e39d3b640 10 monclient: init 2024-03-29T11:44:50.963 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.957+0000 7f8e39d3b640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:50.963 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.957+0000 7f8e39d3b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:44:50.963 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.957+0000 7f8e39d3b640 10 monclient(hunting): picked mon.noname-b con 0x7f8e340b4330 addr v1:172.21.15.72:6790/0 2024-03-29T11:44:50.963 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.957+0000 7f8e39d3b640 10 monclient(hunting): picked mon.noname-a con 0x7f8e341470b0 addr v1:172.21.15.72:6789/0 2024-03-29T11:44:50.963 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.957+0000 7f8e39d3b640 10 monclient(hunting): picked mon.noname-c con 0x7f8e341439b0 addr v1:172.21.15.89:6789/0 2024-03-29T11:44:50.964 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.957+0000 7f8e39d3b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e3406f2a0 con 0x7f8e341470b0 2024-03-29T11:44:50.964 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.957+0000 7f8e39d3b640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e34146b90 con 0x7f8e341439b0 2024-03-29T11:44:50.964 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.957+0000 7f8e39d3b640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e340b2ca0 con 0x7f8e340b4330 2024-03-29T11:44:50.964 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.957+0000 7f8e39d3b640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:50.964 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.957+0000 7f8e39d3b640 10 monclient(hunting): authenticate will time out at 1450.245825s 2024-03-29T11:44:50.964 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.958+0000 7f8e337fe640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f8e340b4330 0x7f8e340b4710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:37234/0 (socket says 172.21.15.72:37234) 2024-03-29T11:44:50.964 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.958+0000 7f8e337fe640 1 -- 172.21.15.72:0/1667284814 learned_addr learned my addr 172.21.15.72:0/1667284814 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:44:50.964 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.958+0000 7f8e327fc640 1 -- 172.21.15.72:0/1667284814 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 250516725 0 0) 0x7f8e340b2ca0 con 0x7f8e340b4330 2024-03-29T11:44:50.964 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.958+0000 7f8e327fc640 10 monclient(hunting): _negotiate 2024-03-29T11:44:50.964 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.958+0000 7f8e327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:50.964 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.958+0000 7f8e327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:50.964 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.958+0000 7f8e327fc640 10 monclient(hunting): my global_id is 4595 2024-03-29T11:44:50.964 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.958+0000 7f8e327fc640 1 -- 172.21.15.72:0/1667284814 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e1c003520 con 0x7f8e340b4330 2024-03-29T11:44:50.964 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.958+0000 7f8e327fc640 1 -- 172.21.15.72:0/1667284814 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4074163131 0 0) 0x7f8e1c003520 con 0x7f8e340b4330 2024-03-29T11:44:50.964 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.958+0000 7f8e327fc640 1 -- 172.21.15.72:0/1667284814 >> v1:172.21.15.89:6789/0 conn(0x7f8e341439b0 legacy=0x7f8e34151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:44:50.965 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.958+0000 7f8e327fc640 1 -- 172.21.15.72:0/1667284814 >> v1:172.21.15.72:6789/0 conn(0x7f8e341470b0 legacy=0x7f8e340b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:44:50.966 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.960+0000 7f8e327fc640 10 monclient: _finish_hunting 0 2024-03-29T11:44:50.966 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.960+0000 7f8e327fc640 1 monclient: found mon.noname-b 2024-03-29T11:44:50.966 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.960+0000 7f8e327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:50.966 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.960+0000 7f8e327fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.72:6790/0 2024-03-29T11:44:50.966 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.960+0000 7f8e327fc640 1 -- 172.21.15.72:0/1667284814 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e3406e0c0 con 0x7f8e340b4330 2024-03-29T11:44:50.966 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.960+0000 7f8e327fc640 10 monclient: _finish_auth 0 2024-03-29T11:44:50.966 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.960+0000 7f8e327fc640 10 monclient: _check_auth_tickets 2024-03-29T11:44:50.966 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.960+0000 7f8e327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:50.966 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.960+0000 7f8e327fc640 1 -- 172.21.15.72:0/1667284814 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f8e24001690 con 0x7f8e340b4330 2024-03-29T11:44:50.967 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.960+0000 7f8e327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:50.967 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.960+0000 7f8e327fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:50.967 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.960+0000 7f8e327fc640 10 monclient: dump: 2024-03-29T11:44:50.967 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:44:50.967 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:44:50.967 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:50.967 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:50.967 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:50.967 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:50.967 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:44:50.967 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:44:50.967 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:44:50.967 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:50.967 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.961+0000 7f8e39d3b640 5 monclient: authenticate success, global_id 4595 2024-03-29T11:44:50.967 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.961+0000 7f8e39d3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:50.967 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.961+0000 7f8e327fc640 1 -- 172.21.15.72:0/1667284814 <== mon.2 v1:172.21.15.72:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e24001860 con 0x7f8e340b4330 2024-03-29T11:44:50.968 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.961+0000 7f8e327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:50.968 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.961+0000 7f8e327fc640 1 -- 172.21.15.72:0/1667284814 <== mon.2 v1:172.21.15.72:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f8e24001cb0 con 0x7f8e340b4330 2024-03-29T11:44:50.968 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.961+0000 7f8e39d3b640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:50.968 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.961+0000 7f8e39d3b640 4 set_mon_vals no callback set 2024-03-29T11:44:50.968 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.961+0000 7f8e327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:50.968 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.961+0000 7f8e327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:50.968 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.961+0000 7f8e327fc640 10 monclient: dump: 2024-03-29T11:44:50.968 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:44:50.968 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:44:50.968 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:50.968 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:50.968 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:50.968 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:50.968 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:44:50.968 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:44:50.968 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:44:50.969 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:50.969 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.962+0000 7f8e39d3b640 10 monclient: shutdown 2024-03-29T11:44:50.969 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.962+0000 7f8e39d3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:50.969 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.962+0000 7f8e39d3b640 1 -- 172.21.15.72:0/1667284814 >> v1:172.21.15.72:6790/0 conn(0x7f8e340b4330 legacy=0x7f8e340b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:50.969 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.962+0000 7f8e39d3b640 1 -- 172.21.15.72:0/1667284814 shutdown_connections 2024-03-29T11:44:50.969 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.962+0000 7f8e39d3b640 1 -- 172.21.15.72:0/1667284814 >> 172.21.15.72:0/1667284814 conn(0x7f8e34066a00 msgr2=0x7f8e34066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:50.969 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.962+0000 7f8e39d3b640 1 -- 172.21.15.72:0/1667284814 shutdown_connections 2024-03-29T11:44:50.969 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.962+0000 7f8e39d3b640 1 -- 172.21.15.72:0/1667284814 wait complete. 2024-03-29T11:44:50.976 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.963+0000 7f8e39d3b640 10 monclient: build_initial_monmap 2024-03-29T11:44:50.976 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.963+0000 7f8e39d3b640 10 monclient: monmap: 2024-03-29T11:44:50.977 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:44:50.977 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:50.977 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:44:50.977 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:44:50.977 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:50.977 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:50.977 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:44:50.977 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:44:50.977 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:44:50.977 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:50.977 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.963+0000 7f8e39d3b640 1 librados: starting msgr at 2024-03-29T11:44:50.977 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.963+0000 7f8e39d3b640 1 librados: starting objecter 2024-03-29T11:44:50.977 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.963+0000 7f8e39d3b640 1 Processor -- start 2024-03-29T11:44:50.978 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.964+0000 7f8e39d3b640 1 -- start start 2024-03-29T11:44:50.978 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.964+0000 7f8e39d3b640 1 librados: setting wanted keys 2024-03-29T11:44:50.978 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.964+0000 7f8e39d3b640 1 librados: calling monclient init 2024-03-29T11:44:50.978 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.964+0000 7f8e39d3b640 10 monclient: init 2024-03-29T11:44:50.978 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.964+0000 7f8e39d3b640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:50.978 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.964+0000 7f8e39d3b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:44:50.978 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.964+0000 7f8e39d3b640 10 monclient(hunting): picked mon.noname-c con 0x7f8e340b4330 addr v1:172.21.15.72:6790/0 2024-03-29T11:44:50.979 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.964+0000 7f8e39d3b640 10 monclient(hunting): picked mon.noname-b con 0x7f8e341439b0 addr v1:172.21.15.89:6789/0 2024-03-29T11:44:50.979 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.964+0000 7f8e39d3b640 10 monclient(hunting): picked mon.noname-a con 0x7f8e341470b0 addr v1:172.21.15.72:6789/0 2024-03-29T11:44:50.979 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.964+0000 7f8e39d3b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e341db4b0 con 0x7f8e341470b0 2024-03-29T11:44:50.979 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.964+0000 7f8e39d3b640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e341dc330 con 0x7f8e341439b0 2024-03-29T11:44:50.979 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.964+0000 7f8e39d3b640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e341db700 con 0x7f8e340b4330 2024-03-29T11:44:50.979 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.964+0000 7f8e39d3b640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:50.979 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.964+0000 7f8e39d3b640 10 monclient(hunting): authenticate will time out at 1450.252858s 2024-03-29T11:44:50.979 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.964+0000 7f8e337fe640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f8e340b4330 0x7f8e341dbc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:37824/0 (socket says 172.21.15.72:37824) 2024-03-29T11:44:50.979 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.965+0000 7f8e337fe640 1 -- 172.21.15.72:0/1062750015 learned_addr learned my addr 172.21.15.72:0/1062750015 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:44:50.979 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.965+0000 7f8e1bfff640 10 client.?.objecter ms_handle_connect 0x7f8e340b4330 2024-03-29T11:44:50.980 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.965+0000 7f8e1bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:50.980 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.965+0000 7f8e1bfff640 1 -- 172.21.15.72:0/1062750015 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3434432989 0 0) 0x7f8e341db700 con 0x7f8e340b4330 2024-03-29T11:44:50.980 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.965+0000 7f8e1bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:44:50.980 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.965+0000 7f8e1bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:50.980 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.965+0000 7f8e1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:50.980 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.965+0000 7f8e1bfff640 10 monclient(hunting): my global_id is 4598 2024-03-29T11:44:50.980 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.965+0000 7f8e1bfff640 1 -- 172.21.15.72:0/1062750015 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e20004eb0 con 0x7f8e340b4330 2024-03-29T11:44:50.980 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.966+0000 7f8e1bfff640 10 client.?.objecter ms_handle_connect 0x7f8e341439b0 2024-03-29T11:44:50.980 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.966+0000 7f8e1bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:50.980 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.966+0000 7f8e1bfff640 1 -- 172.21.15.72:0/1062750015 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3246544369 0 0) 0x7f8e341dc330 con 0x7f8e341439b0 2024-03-29T11:44:50.980 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.966+0000 7f8e1bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:44:50.980 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.966+0000 7f8e1bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:50.980 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.966+0000 7f8e1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:50.980 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.966+0000 7f8e1bfff640 10 monclient(hunting): my global_id is 4522 2024-03-29T11:44:50.981 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.966+0000 7f8e1bfff640 1 -- 172.21.15.72:0/1062750015 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e341db700 con 0x7f8e341439b0 2024-03-29T11:44:50.981 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.966+0000 7f8e1bfff640 1 -- 172.21.15.72:0/1062750015 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2776471118 0 0) 0x7f8e20004eb0 con 0x7f8e340b4330 2024-03-29T11:44:50.981 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.966+0000 7f8e1bfff640 1 -- 172.21.15.72:0/1062750015 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e341dc330 con 0x7f8e340b4330 2024-03-29T11:44:50.981 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.966+0000 7f8e1bfff640 1 -- 172.21.15.72:0/1062750015 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f8e24001c30 con 0x7f8e340b4330 2024-03-29T11:44:50.981 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.966+0000 7f8e1bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:50.981 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.966+0000 7f8e1bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:44:50.981 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.966+0000 7f8e1bfff640 10 monclient(hunting): dump: 2024-03-29T11:44:50.981 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:44:50.981 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:44:50.981 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:50.981 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:50.982 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:50.982 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:50.982 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:44:50.982 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:44:50.982 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:44:50.982 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:50.982 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.966+0000 7f8e1bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:50.982 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.967+0000 7f8e1bfff640 1 -- 172.21.15.72:0/1062750015 <== mon.2 v1:172.21.15.72:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1773151859 0 0) 0x7f8e341dc330 con 0x7f8e340b4330 2024-03-29T11:44:50.982 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.967+0000 7f8e1bfff640 1 -- 172.21.15.72:0/1062750015 >> v1:172.21.15.89:6789/0 conn(0x7f8e341439b0 legacy=0x7f8e341403d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:50.982 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.967+0000 7f8e1bfff640 1 -- 172.21.15.72:0/1062750015 >> v1:172.21.15.72:6789/0 conn(0x7f8e341470b0 legacy=0x7f8e341e6cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:44:50.982 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.968+0000 7f8e1bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:44:50.982 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.968+0000 7f8e1bfff640 1 monclient: found mon.c 2024-03-29T11:44:50.982 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.968+0000 7f8e1bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:50.983 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.968+0000 7f8e1bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:44:50.983 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.968+0000 7f8e1bfff640 1 -- 172.21.15.72:0/1062750015 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e341415c0 con 0x7f8e340b4330 2024-03-29T11:44:50.983 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.968+0000 7f8e1bfff640 10 monclient: _finish_auth 0 2024-03-29T11:44:50.983 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.968+0000 7f8e1bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:44:50.983 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.968+0000 7f8e1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:50.983 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.968+0000 7f8e39d3b640 5 monclient: authenticate success, global_id 4598 2024-03-29T11:44:50.983 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.968+0000 7f8e39d3b640 10 monclient: _renew_subs 2024-03-29T11:44:50.983 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.968+0000 7f8e39d3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:44:50.983 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.968+0000 7f8e39d3b640 1 -- 172.21.15.72:0/1062750015 --> v1:172.21.15.72:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e34146290 con 0x7f8e340b4330 2024-03-29T11:44:50.983 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.968+0000 7f8e39d3b640 10 client.4598.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:50.983 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.969+0000 7f8e39d3b640 10 monclient: _renew_subs 2024-03-29T11:44:50.984 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.969+0000 7f8e39d3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:44:50.984 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.969+0000 7f8e39d3b640 1 -- 172.21.15.72:0/1062750015 --> v1:172.21.15.72:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e340b7280 con 0x7f8e340b4330 2024-03-29T11:44:50.984 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.969+0000 7f8e39d3b640 1 librados: init done 2024-03-29T11:44:50.984 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.970+0000 7f8e1bfff640 1 -- 172.21.15.72:0/1062750015 <== mon.2 v1:172.21.15.72:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e24004b00 con 0x7f8e340b4330 2024-03-29T11:44:50.984 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.970+0000 7f8e1bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:50.984 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.970+0000 7f8e1bfff640 1 -- 172.21.15.72:0/1062750015 <== mon.2 v1:172.21.15.72:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f8e24004e90 con 0x7f8e340b4330 2024-03-29T11:44:50.984 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.970+0000 7f8e1bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:50.984 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.970+0000 7f8e1bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:50.984 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.970+0000 7f8e1bfff640 10 monclient: dump: 2024-03-29T11:44:50.985 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:44:50.985 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:44:50.985 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:50.985 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:50.985 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:50.985 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:50.985 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:44:50.985 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:44:50.985 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:44:50.985 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:50.986 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.970+0000 7f8e31ffb640 4 set_mon_vals no callback set 2024-03-29T11:44:50.986 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.970+0000 7f8e39d3b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:50.986 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.971+0000 7f8e1bfff640 1 -- 172.21.15.72:0/1062750015 <== mon.2 v1:172.21.15.72:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f8e240050b0 con 0x7f8e340b4330 2024-03-29T11:44:50.986 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.971+0000 7f8e1bfff640 1 -- 172.21.15.72:0/1062750015 <== mon.2 v1:172.21.15.72:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7f8e24084b00 con 0x7f8e340b4330 2024-03-29T11:44:50.986 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.971+0000 7f8e1bfff640 10 client.4598.objecter ms_dispatch 0x7f8e34000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:50.986 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.971+0000 7f8e1bfff640 3 client.4598.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:50.986 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.971+0000 7f8e1bfff640 3 client.4598.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:50.986 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.971+0000 7f8e1bfff640 20 client.4598.objecter dump_active .. 0 homeless 2024-03-29T11:44:50.986 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.972+0000 7f8e39d3b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:50.986 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.972+0000 7f8e39d3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:44:50.986 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.972+0000 7f8e39d3b640 1 -- 172.21.15.72:0/1062750015 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e14005080 con 0x7f8e340b4330 2024-03-29T11:44:50.986 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.972+0000 7f8e32ffd640 10 monclient: get_auth_request con 0x7f8e2006dca0 auth_method 0 2024-03-29T11:44:50.986 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.973+0000 7f8e1bfff640 10 client.4598.objecter ms_handle_connect 0x7f8e2006dca0 2024-03-29T11:44:50.987 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.981+0000 7f8e1bfff640 1 -- 172.21.15.72:0/1062750015 <== mon.2 v1:172.21.15.72:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8e24050440 con 0x7f8e340b4330 2024-03-29T11:44:50.987 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.981+0000 7f8e1bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:50.987 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:50.981+0000 7f8e1bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:51.299 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:51.294+0000 7f8e39d3b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:51.299 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:51.294+0000 7f8e39d3b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:51.299 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:51.294+0000 7f8e39d3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:44:51.299 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:51.294+0000 7f8e39d3b640 1 -- 172.21.15.72:0/1062750015 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e14005330 con 0x7f8e340b4330 2024-03-29T11:44:51.300 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:51.296+0000 7f8e1bfff640 1 -- 172.21.15.72:0/1062750015 <== mon.2 v1:172.21.15.72:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7f8e24052370 con 0x7f8e340b4330 2024-03-29T11:44:51.300 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:51.296+0000 7f8e1bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:51.300 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:51.296+0000 7f8e1bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:51.300 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:44:51.300 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:51.303 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:51.299+0000 7f8e39d3b640 10 librados: watch_flush enter 2024-03-29T11:44:51.303 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:51.299+0000 7f8e39d3b640 10 librados: watch_flush exit 2024-03-29T11:44:51.303 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:51.299+0000 7f8e39d3b640 20 client.4598.objecter shutdown clearing up homeless session... 2024-03-29T11:44:51.303 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:51.299+0000 7f8e39d3b640 10 client.4598.objecter successfully canceled tick 2024-03-29T11:44:51.304 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:51.299+0000 7f8e39d3b640 1 -- 172.21.15.72:0/1062750015 >> v1:172.21.15.72:6816/1578866102 conn(0x7f8e2006dca0 legacy=0x7f8e20070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:51.304 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:51.299+0000 7f8e39d3b640 10 monclient: shutdown 2024-03-29T11:44:51.304 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:51.299+0000 7f8e39d3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:51.304 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:51.299+0000 7f8e39d3b640 1 -- 172.21.15.72:0/1062750015 >> v1:172.21.15.72:6790/0 conn(0x7f8e340b4330 legacy=0x7f8e341dbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:51.304 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:51.299+0000 7f8e39d3b640 1 -- 172.21.15.72:0/1062750015 shutdown_connections 2024-03-29T11:44:51.304 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:51.299+0000 7f8e39d3b640 1 -- 172.21.15.72:0/1062750015 >> 172.21.15.72:0/1062750015 conn(0x7f8e34066a00 msgr2=0x7f8e341d1470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:51.304 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:51.300+0000 7f8e39d3b640 1 -- 172.21.15.72:0/1062750015 shutdown_connections 2024-03-29T11:44:51.304 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:51.300+0000 7f8e39d3b640 1 -- 172.21.15.72:0/1062750015 wait complete. 2024-03-29T11:44:51.305 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:51.300+0000 7f8e39d3b640 1 librados: shutdown 2024-03-29T11:44:51.319 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:58.127 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:58.259 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.254+0000 7fbb62938640 10 monclient: get_monmap_and_config 2024-03-29T11:44:58.259 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.254+0000 7fbb62938640 10 monclient: build_initial_monmap 2024-03-29T11:44:58.259 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.254+0000 7fbb62938640 10 monclient: monmap: 2024-03-29T11:44:58.259 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:44:58.259 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:44:58.259 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:44:58.255760+0000 2024-03-29T11:44:58.259 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:44:58.255760+0000 2024-03-29T11:44:58.259 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:58.259 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:58.259 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:44:58.259 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:44:58.259 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:44:58.260 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:58.261 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.255+0000 7fbb62938640 1 Processor -- start 2024-03-29T11:44:58.261 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.256+0000 7fbb62938640 1 -- start start 2024-03-29T11:44:58.261 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.256+0000 7fbb62938640 10 monclient: init 2024-03-29T11:44:58.261 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.256+0000 7fbb62938640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:58.261 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.257+0000 7fbb62938640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:44:58.261 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.257+0000 7fbb62938640 10 monclient(hunting): picked mon.noname-a con 0x7fbb5c1457f0 addr v1:172.21.15.72:6789/0 2024-03-29T11:44:58.261 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.257+0000 7fbb62938640 10 monclient(hunting): picked mon.noname-c con 0x7fbb5c13f650 addr v1:172.21.15.89:6789/0 2024-03-29T11:44:58.261 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.257+0000 7fbb62938640 10 monclient(hunting): picked mon.noname-b con 0x7fbb5c0a9d70 addr v1:172.21.15.72:6790/0 2024-03-29T11:44:58.261 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.257+0000 7fbb62938640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb5c0aaef0 con 0x7fbb5c1457f0 2024-03-29T11:44:58.262 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.257+0000 7fbb62938640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb5c145bd0 con 0x7fbb5c13f650 2024-03-29T11:44:58.262 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.257+0000 7fbb62938640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb5c145eb0 con 0x7fbb5c0a9d70 2024-03-29T11:44:58.262 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.257+0000 7fbb62938640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:58.262 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.257+0000 7fbb62938640 10 monclient(hunting): authenticate will time out at 1457.545291s 2024-03-29T11:44:58.263 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.258+0000 7fbb5bfff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fbb5c1457f0 0x7fbb5c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:58160/0 (socket says 172.21.15.72:58160) 2024-03-29T11:44:58.263 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.258+0000 7fbb5bfff640 1 -- 172.21.15.72:0/1001418353 learned_addr learned my addr 172.21.15.72:0/1001418353 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:44:58.263 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.258+0000 7fbb5affd640 1 -- 172.21.15.72:0/1001418353 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2569038577 0 0) 0x7fbb5c0aaef0 con 0x7fbb5c1457f0 2024-03-29T11:44:58.263 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.258+0000 7fbb5affd640 10 monclient(hunting): _negotiate 2024-03-29T11:44:58.263 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.258+0000 7fbb5affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:58.263 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.258+0000 7fbb5affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:58.263 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.258+0000 7fbb5affd640 10 monclient(hunting): my global_id is 4569 2024-03-29T11:44:58.263 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.258+0000 7fbb5affd640 1 -- 172.21.15.72:0/1001418353 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb44003520 con 0x7fbb5c1457f0 2024-03-29T11:44:58.263 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.258+0000 7fbb5affd640 1 -- 172.21.15.72:0/1001418353 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2154575192 0 0) 0x7fbb44003520 con 0x7fbb5c1457f0 2024-03-29T11:44:58.263 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.258+0000 7fbb5affd640 1 -- 172.21.15.72:0/1001418353 >> v1:172.21.15.72:6790/0 conn(0x7fbb5c0a9d70 legacy=0x7fbb5c14d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:44:58.264 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.259+0000 7fbb5affd640 1 -- 172.21.15.72:0/1001418353 >> v1:172.21.15.89:6789/0 conn(0x7fbb5c13f650 legacy=0x7fbb5c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:58.264 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.259+0000 7fbb5affd640 10 monclient: _finish_hunting 0 2024-03-29T11:44:58.264 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.259+0000 7fbb5affd640 1 monclient: found mon.noname-a 2024-03-29T11:44:58.264 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.259+0000 7fbb5affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:58.264 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.259+0000 7fbb5affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.72:6789/0 2024-03-29T11:44:58.264 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.259+0000 7fbb5affd640 1 -- 172.21.15.72:0/1001418353 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb5c146340 con 0x7fbb5c1457f0 2024-03-29T11:44:58.264 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.259+0000 7fbb5affd640 10 monclient: _finish_auth 0 2024-03-29T11:44:58.265 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.259+0000 7fbb5affd640 10 monclient: _check_auth_tickets 2024-03-29T11:44:58.265 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.259+0000 7fbb5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:58.265 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.259+0000 7fbb5affd640 1 -- 172.21.15.72:0/1001418353 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fbb50001670 con 0x7fbb5c1457f0 2024-03-29T11:44:58.265 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.259+0000 7fbb5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:58.265 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.259+0000 7fbb5affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:44:58.265 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.259+0000 7fbb5affd640 10 monclient: dump: 2024-03-29T11:44:58.265 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:44:58.265 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:44:58.265 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:58.265 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:58.265 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:58.265 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:58.265 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:44:58.265 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:44:58.265 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:44:58.266 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:58.266 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.259+0000 7fbb62938640 5 monclient: authenticate success, global_id 4569 2024-03-29T11:44:58.266 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.259+0000 7fbb62938640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:58.266 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.260+0000 7fbb5affd640 1 -- 172.21.15.72:0/1001418353 <== mon.0 v1:172.21.15.72:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb500018d0 con 0x7fbb5c1457f0 2024-03-29T11:44:58.266 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.260+0000 7fbb5affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:58.266 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.260+0000 7fbb5affd640 1 -- 172.21.15.72:0/1001418353 <== mon.0 v1:172.21.15.72:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fbb50001d20 con 0x7fbb5c1457f0 2024-03-29T11:44:58.266 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.260+0000 7fbb5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:58.266 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.260+0000 7fbb5affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:58.266 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.260+0000 7fbb5affd640 10 monclient: dump: 2024-03-29T11:44:58.266 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:44:58.266 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:44:58.266 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:58.266 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:58.266 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:58.266 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:58.267 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:44:58.267 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:44:58.267 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:44:58.267 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:58.267 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.260+0000 7fbb62938640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:58.267 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.260+0000 7fbb62938640 4 set_mon_vals no callback set 2024-03-29T11:44:58.268 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.261+0000 7fbb62938640 10 monclient: shutdown 2024-03-29T11:44:58.268 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.261+0000 7fbb62938640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:58.269 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.261+0000 7fbb62938640 1 -- 172.21.15.72:0/1001418353 >> v1:172.21.15.72:6789/0 conn(0x7fbb5c1457f0 legacy=0x7fbb5c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:58.269 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.261+0000 7fbb62938640 1 -- 172.21.15.72:0/1001418353 shutdown_connections 2024-03-29T11:44:58.269 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.261+0000 7fbb62938640 1 -- 172.21.15.72:0/1001418353 >> 172.21.15.72:0/1001418353 conn(0x7fbb5c066a00 msgr2=0x7fbb5c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:58.269 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.261+0000 7fbb62938640 1 -- 172.21.15.72:0/1001418353 shutdown_connections 2024-03-29T11:44:58.269 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.261+0000 7fbb62938640 1 -- 172.21.15.72:0/1001418353 wait complete. 2024-03-29T11:44:58.269 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.262+0000 7fbb62938640 10 monclient: build_initial_monmap 2024-03-29T11:44:58.269 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.262+0000 7fbb62938640 10 monclient: monmap: 2024-03-29T11:44:58.269 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:44:58.269 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:58.269 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:44:58.269 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:44:58.269 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:58.269 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:58.269 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:44:58.269 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:44:58.270 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:44:58.270 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:58.270 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.263+0000 7fbb62938640 1 librados: starting msgr at 2024-03-29T11:44:58.270 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.263+0000 7fbb62938640 1 librados: starting objecter 2024-03-29T11:44:58.270 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.263+0000 7fbb62938640 1 Processor -- start 2024-03-29T11:44:58.270 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.263+0000 7fbb62938640 1 -- start start 2024-03-29T11:44:58.270 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.263+0000 7fbb62938640 1 librados: setting wanted keys 2024-03-29T11:44:58.270 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.263+0000 7fbb62938640 1 librados: calling monclient init 2024-03-29T11:44:58.270 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.263+0000 7fbb62938640 10 monclient: init 2024-03-29T11:44:58.270 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.264+0000 7fbb62938640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:58.270 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.264+0000 7fbb62938640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:44:58.272 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.265+0000 7fbb62938640 10 monclient(hunting): picked mon.noname-c con 0x7fbb5c0a9d70 addr v1:172.21.15.72:6790/0 2024-03-29T11:44:58.272 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.265+0000 7fbb62938640 10 monclient(hunting): picked mon.noname-b con 0x7fbb5c13f650 addr v1:172.21.15.89:6789/0 2024-03-29T11:44:58.272 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.265+0000 7fbb62938640 10 monclient(hunting): picked mon.noname-a con 0x7fbb5c1457f0 addr v1:172.21.15.72:6789/0 2024-03-29T11:44:58.272 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.265+0000 7fbb62938640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb5c142c50 con 0x7fbb5c1457f0 2024-03-29T11:44:58.273 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.265+0000 7fbb62938640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb5c145160 con 0x7fbb5c13f650 2024-03-29T11:44:58.273 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.265+0000 7fbb62938640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb5c1452f0 con 0x7fbb5c0a9d70 2024-03-29T11:44:58.273 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.265+0000 7fbb62938640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:58.273 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.265+0000 7fbb62938640 10 monclient(hunting): authenticate will time out at 1457.553779s 2024-03-29T11:44:58.273 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.265+0000 7fbb60ed5640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fbb5c1457f0 0x7fbb5c143870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:59136/0 (socket says 172.21.15.72:59136) 2024-03-29T11:44:58.273 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.265+0000 7fbb60ed5640 1 -- 172.21.15.72:0/3253393058 learned_addr learned my addr 172.21.15.72:0/3253393058 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:44:58.273 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.266+0000 7fbb58ff9640 10 client.?.objecter ms_handle_connect 0x7fbb5c1457f0 2024-03-29T11:44:58.273 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.266+0000 7fbb58ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:58.273 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.266+0000 7fbb58ff9640 10 client.?.objecter ms_handle_connect 0x7fbb5c13f650 2024-03-29T11:44:58.273 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.266+0000 7fbb58ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:58.273 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.266+0000 7fbb58ff9640 1 -- 172.21.15.72:0/3253393058 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1415835922 0 0) 0x7fbb5c142c50 con 0x7fbb5c1457f0 2024-03-29T11:44:58.273 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.266+0000 7fbb58ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:44:58.273 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.266+0000 7fbb58ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:58.273 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.266+0000 7fbb58ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:58.273 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.266+0000 7fbb58ff9640 10 monclient(hunting): my global_id is 4572 2024-03-29T11:44:58.274 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.266+0000 7fbb58ff9640 1 -- 172.21.15.72:0/3253393058 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb48004eb0 con 0x7fbb5c1457f0 2024-03-29T11:44:58.274 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.266+0000 7fbb58ff9640 1 -- 172.21.15.72:0/3253393058 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1144759267 0 0) 0x7fbb5c145160 con 0x7fbb5c13f650 2024-03-29T11:44:58.274 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.266+0000 7fbb58ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:44:58.274 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.266+0000 7fbb58ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:58.274 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.266+0000 7fbb58ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:58.274 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.266+0000 7fbb58ff9640 10 monclient(hunting): my global_id is 4528 2024-03-29T11:44:58.274 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.266+0000 7fbb58ff9640 1 -- 172.21.15.72:0/3253393058 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb5c142c50 con 0x7fbb5c13f650 2024-03-29T11:44:58.274 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.266+0000 7fbb58ff9640 1 -- 172.21.15.72:0/3253393058 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1469556916 0 0) 0x7fbb48004eb0 con 0x7fbb5c1457f0 2024-03-29T11:44:58.274 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.266+0000 7fbb58ff9640 1 -- 172.21.15.72:0/3253393058 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb5c145160 con 0x7fbb5c1457f0 2024-03-29T11:44:58.274 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.266+0000 7fbb58ff9640 1 -- 172.21.15.72:0/3253393058 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fbb54001c60 con 0x7fbb5c1457f0 2024-03-29T11:44:58.274 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.266+0000 7fbb58ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:58.274 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.266+0000 7fbb58ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:44:58.274 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.266+0000 7fbb58ff9640 10 monclient(hunting): dump: 2024-03-29T11:44:58.274 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:44:58.274 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:44:58.275 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:58.275 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:58.275 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:58.275 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:58.275 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:44:58.275 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:44:58.275 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:44:58.275 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:58.275 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.266+0000 7fbb58ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:58.275 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.266+0000 7fbb58ff9640 1 -- 172.21.15.72:0/3253393058 <== mon.0 v1:172.21.15.72:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4270145598 0 0) 0x7fbb5c145160 con 0x7fbb5c1457f0 2024-03-29T11:44:58.275 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.266+0000 7fbb58ff9640 1 -- 172.21.15.72:0/3253393058 >> v1:172.21.15.72:6790/0 conn(0x7fbb5c0a9d70 legacy=0x7fbb5c134f60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:44:58.275 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.268+0000 7fbb58ff9640 1 -- 172.21.15.72:0/3253393058 >> v1:172.21.15.89:6789/0 conn(0x7fbb5c13f650 legacy=0x7fbb5c143160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:58.275 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.268+0000 7fbb58ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:44:58.275 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.268+0000 7fbb58ff9640 1 monclient: found mon.a 2024-03-29T11:44:58.275 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.268+0000 7fbb58ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:58.275 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.268+0000 7fbb58ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:44:58.276 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.268+0000 7fbb58ff9640 1 -- 172.21.15.72:0/3253393058 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb5c144ca0 con 0x7fbb5c1457f0 2024-03-29T11:44:58.276 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.268+0000 7fbb58ff9640 10 monclient: _finish_auth 0 2024-03-29T11:44:58.276 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.268+0000 7fbb58ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:44:58.276 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.268+0000 7fbb58ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:58.276 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.268+0000 7fbb62938640 5 monclient: authenticate success, global_id 4572 2024-03-29T11:44:58.276 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.268+0000 7fbb62938640 10 monclient: _renew_subs 2024-03-29T11:44:58.276 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.268+0000 7fbb62938640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:44:58.276 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.268+0000 7fbb62938640 1 -- 172.21.15.72:0/3253393058 --> v1:172.21.15.72:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb5c144e90 con 0x7fbb5c1457f0 2024-03-29T11:44:58.276 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.268+0000 7fbb62938640 10 client.4572.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:58.276 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.268+0000 7fbb62938640 10 monclient: _renew_subs 2024-03-29T11:44:58.276 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.268+0000 7fbb62938640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:44:58.276 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.268+0000 7fbb62938640 1 -- 172.21.15.72:0/3253393058 --> v1:172.21.15.72:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb5c142870 con 0x7fbb5c1457f0 2024-03-29T11:44:58.276 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.268+0000 7fbb62938640 1 librados: init done 2024-03-29T11:44:58.278 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.269+0000 7fbb62938640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:58.278 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.269+0000 7fbb62938640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:58.278 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.269+0000 7fbb62938640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:44:58.279 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.269+0000 7fbb62938640 1 -- 172.21.15.72:0/3253393058 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb5c075f10 con 0x7fbb5c1457f0 2024-03-29T11:44:58.279 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.269+0000 7fbb58ff9640 1 -- 172.21.15.72:0/3253393058 <== mon.0 v1:172.21.15.72:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb54003970 con 0x7fbb5c1457f0 2024-03-29T11:44:58.279 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.270+0000 7fbb58ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:58.279 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.270+0000 7fbb58ff9640 1 -- 172.21.15.72:0/3253393058 <== mon.0 v1:172.21.15.72:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fbb54003d40 con 0x7fbb5c1457f0 2024-03-29T11:44:58.279 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.270+0000 7fbb58ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:58.279 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.270+0000 7fbb58ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:58.279 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.270+0000 7fbb58ff9640 10 monclient: dump: 2024-03-29T11:44:58.279 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:44:58.279 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:44:58.279 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:58.279 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:44:58.279 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:58.279 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:44:58.279 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:44:58.279 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:44:58.279 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:44:58.280 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:44:58.280 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.270+0000 7fbb5a7fc640 4 set_mon_vals no callback set 2024-03-29T11:44:58.280 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.271+0000 7fbb58ff9640 1 -- 172.21.15.72:0/3253393058 <== mon.0 v1:172.21.15.72:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7fbb54003f60 con 0x7fbb5c1457f0 2024-03-29T11:44:58.280 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.271+0000 7fbb58ff9640 1 -- 172.21.15.72:0/3253393058 <== mon.0 v1:172.21.15.72:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1952735773 0 0) 0x7fbb54084d90 con 0x7fbb5c1457f0 2024-03-29T11:44:58.280 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.271+0000 7fbb58ff9640 10 client.4572.objecter ms_dispatch 0x7fbb5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:58.280 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.271+0000 7fbb58ff9640 3 client.4572.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:58.280 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.271+0000 7fbb58ff9640 3 client.4572.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:58.280 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.271+0000 7fbb5bfff640 10 monclient: get_auth_request con 0x7fbb4806da80 auth_method 0 2024-03-29T11:44:58.280 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.271+0000 7fbb58ff9640 20 client.4572.objecter dump_active .. 0 homeless 2024-03-29T11:44:58.280 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.272+0000 7fbb58ff9640 10 client.4572.objecter ms_handle_connect 0x7fbb4806da80 2024-03-29T11:44:58.283 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.278+0000 7fbb58ff9640 1 -- 172.21.15.72:0/3253393058 <== mon.0 v1:172.21.15.72:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbb54050670 con 0x7fbb5c1457f0 2024-03-29T11:44:58.283 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.278+0000 7fbb58ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:58.283 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.278+0000 7fbb58ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:58.594 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.590+0000 7fbb62938640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:58.594 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.590+0000 7fbb62938640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:58.594 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.590+0000 7fbb62938640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:44:58.594 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.590+0000 7fbb62938640 1 -- 172.21.15.72:0/3253393058 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb5c0b3680 con 0x7fbb5c1457f0 2024-03-29T11:44:58.595 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.591+0000 7fbb58ff9640 1 -- 172.21.15.72:0/3253393058 <== mon.0 v1:172.21.15.72:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1227915879) 0x7fbb540525a0 con 0x7fbb5c1457f0 2024-03-29T11:44:58.595 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.591+0000 7fbb58ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:58.595 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.591+0000 7fbb58ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:58.595 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:44:58.595 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:35:02.347428+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:02.347428+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:58.598 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.594+0000 7fbb62938640 10 librados: watch_flush enter 2024-03-29T11:44:58.598 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.594+0000 7fbb62938640 10 librados: watch_flush exit 2024-03-29T11:44:58.598 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.594+0000 7fbb62938640 20 client.4572.objecter shutdown clearing up homeless session... 2024-03-29T11:44:58.598 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.594+0000 7fbb62938640 10 client.4572.objecter successfully canceled tick 2024-03-29T11:44:58.600 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.596+0000 7fbb62938640 1 -- 172.21.15.72:0/3253393058 >> v1:172.21.15.72:6816/1578866102 conn(0x7fbb4806da80 legacy=0x7fbb4806ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:58.600 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.596+0000 7fbb62938640 10 monclient: shutdown 2024-03-29T11:44:58.600 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.596+0000 7fbb62938640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:58.600 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.596+0000 7fbb62938640 1 -- 172.21.15.72:0/3253393058 >> v1:172.21.15.72:6789/0 conn(0x7fbb5c1457f0 legacy=0x7fbb5c143870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:58.600 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.596+0000 7fbb62938640 1 -- 172.21.15.72:0/3253393058 shutdown_connections 2024-03-29T11:44:58.601 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.596+0000 7fbb62938640 1 -- 172.21.15.72:0/3253393058 >> 172.21.15.72:0/3253393058 conn(0x7fbb5c066a00 msgr2=0x7fbb5c0b4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:58.601 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.596+0000 7fbb62938640 1 -- 172.21.15.72:0/3253393058 shutdown_connections 2024-03-29T11:44:58.601 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.596+0000 7fbb62938640 1 -- 172.21.15.72:0/3253393058 wait complete. 2024-03-29T11:44:58.601 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:44:58.596+0000 7fbb62938640 1 librados: shutdown 2024-03-29T11:44:58.616 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:05.424 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:05.572 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.562+0000 7f2d43040640 10 monclient: get_monmap_and_config 2024-03-29T11:45:05.572 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.562+0000 7f2d43040640 10 monclient: build_initial_monmap 2024-03-29T11:45:05.572 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.562+0000 7f2d43040640 10 monclient: monmap: 2024-03-29T11:45:05.572 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:45:05.572 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:45:05.572 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:45:05.563629+0000 2024-03-29T11:45:05.572 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:45:05.563629+0000 2024-03-29T11:45:05.572 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:05.573 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:05.573 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:45:05.573 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:45:05.573 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:45:05.573 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:05.573 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.564+0000 7f2d43040640 1 Processor -- start 2024-03-29T11:45:05.573 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.564+0000 7f2d43040640 1 -- start start 2024-03-29T11:45:05.573 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.564+0000 7f2d43040640 10 monclient: init 2024-03-29T11:45:05.573 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.565+0000 7f2d43040640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:05.573 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.565+0000 7f2d43040640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:45:05.573 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.565+0000 7f2d43040640 10 monclient(hunting): picked mon.noname-b con 0x7f2d3c0ad6c0 addr v1:172.21.15.72:6790/0 2024-03-29T11:45:05.573 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.565+0000 7f2d43040640 10 monclient(hunting): picked mon.noname-a con 0x7f2d3c0af6c0 addr v1:172.21.15.72:6789/0 2024-03-29T11:45:05.573 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.565+0000 7f2d43040640 10 monclient(hunting): picked mon.noname-c con 0x7f2d3c14fdc0 addr v1:172.21.15.89:6789/0 2024-03-29T11:45:05.573 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.565+0000 7f2d43040640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d3c0a94e0 con 0x7f2d3c0af6c0 2024-03-29T11:45:05.573 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.565+0000 7f2d43040640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d3c0af3a0 con 0x7f2d3c14fdc0 2024-03-29T11:45:05.574 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.565+0000 7f2d43040640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d3c0afb60 con 0x7f2d3c0ad6c0 2024-03-29T11:45:05.574 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.565+0000 7f2d43040640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:05.574 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.565+0000 7f2d43040640 10 monclient(hunting): authenticate will time out at 1464.853191s 2024-03-29T11:45:05.574 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.565+0000 7f2d40ddc640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f2d3c0ad6c0 0x7f2d3c14f6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:53734/0 (socket says 172.21.15.72:53734) 2024-03-29T11:45:05.574 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.565+0000 7f2d40ddc640 1 -- 172.21.15.72:0/2045025032 learned_addr learned my addr 172.21.15.72:0/2045025032 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:45:05.574 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.567+0000 7f2d3b7fe640 1 -- 172.21.15.72:0/2045025032 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2632417643 0 0) 0x7f2d3c0afb60 con 0x7f2d3c0ad6c0 2024-03-29T11:45:05.574 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.567+0000 7f2d3b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:45:05.574 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.567+0000 7f2d3b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:05.574 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.567+0000 7f2d3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:05.574 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.567+0000 7f2d3b7fe640 10 monclient(hunting): my global_id is 4601 2024-03-29T11:45:05.574 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.567+0000 7f2d3b7fe640 1 -- 172.21.15.72:0/2045025032 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d24003520 con 0x7f2d3c0ad6c0 2024-03-29T11:45:05.574 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.568+0000 7f2d3b7fe640 1 -- 172.21.15.72:0/2045025032 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2950658288 0 0) 0x7f2d24003520 con 0x7f2d3c0ad6c0 2024-03-29T11:45:05.574 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.568+0000 7f2d3b7fe640 1 -- 172.21.15.72:0/2045025032 >> v1:172.21.15.89:6789/0 conn(0x7f2d3c14fdc0 legacy=0x7f2d3c15a8e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:45:05.574 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.568+0000 7f2d3b7fe640 1 -- 172.21.15.72:0/2045025032 >> v1:172.21.15.72:6789/0 conn(0x7f2d3c0af6c0 legacy=0x7f2d3c152980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:45:05.575 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.568+0000 7f2d3b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:45:05.575 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.568+0000 7f2d3b7fe640 1 monclient: found mon.noname-b 2024-03-29T11:45:05.576 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.568+0000 7f2d3b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:05.576 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.568+0000 7f2d3b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.72:6790/0 2024-03-29T11:45:05.576 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.568+0000 7f2d3b7fe640 1 -- 172.21.15.72:0/2045025032 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d3c0aab80 con 0x7f2d3c0ad6c0 2024-03-29T11:45:05.576 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.568+0000 7f2d3b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:45:05.576 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.568+0000 7f2d3b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:45:05.576 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.568+0000 7f2d3b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:05.576 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.568+0000 7f2d43040640 5 monclient: authenticate success, global_id 4601 2024-03-29T11:45:05.576 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.568+0000 7f2d43040640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:05.576 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.568+0000 7f2d3b7fe640 1 -- 172.21.15.72:0/2045025032 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f2d34001670 con 0x7f2d3c0ad6c0 2024-03-29T11:45:05.576 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.568+0000 7f2d3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:05.576 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.568+0000 7f2d3b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:45:05.576 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.568+0000 7f2d3b7fe640 10 monclient: dump: 2024-03-29T11:45:05.576 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:45:05.576 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:45:05.576 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:05.576 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:05.577 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:05.577 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:05.577 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:45:05.577 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:45:05.577 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:45:05.577 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:05.577 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.568+0000 7f2d43040640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:05.577 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.569+0000 7f2d3b7fe640 1 -- 172.21.15.72:0/2045025032 <== mon.2 v1:172.21.15.72:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d34001840 con 0x7f2d3c0ad6c0 2024-03-29T11:45:05.577 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.569+0000 7f2d3b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:05.577 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.569+0000 7f2d43040640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:05.577 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.569+0000 7f2d43040640 4 set_mon_vals no callback set 2024-03-29T11:45:05.578 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.570+0000 7f2d43040640 10 monclient: shutdown 2024-03-29T11:45:05.578 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.570+0000 7f2d43040640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:05.578 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.570+0000 7f2d43040640 1 -- 172.21.15.72:0/2045025032 >> v1:172.21.15.72:6790/0 conn(0x7f2d3c0ad6c0 legacy=0x7f2d3c14f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:05.578 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.572+0000 7f2d43040640 1 -- 172.21.15.72:0/2045025032 shutdown_connections 2024-03-29T11:45:05.578 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.572+0000 7f2d43040640 1 -- 172.21.15.72:0/2045025032 >> 172.21.15.72:0/2045025032 conn(0x7f2d3c0b35e0 msgr2=0x7f2d3c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:05.579 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.572+0000 7f2d43040640 1 -- 172.21.15.72:0/2045025032 shutdown_connections 2024-03-29T11:45:05.579 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.572+0000 7f2d43040640 1 -- 172.21.15.72:0/2045025032 wait complete. 2024-03-29T11:45:05.583 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.573+0000 7f2d43040640 10 monclient: build_initial_monmap 2024-03-29T11:45:05.583 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.573+0000 7f2d43040640 10 monclient: monmap: 2024-03-29T11:45:05.583 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:45:05.583 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:05.583 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:45:05.583 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:45:05.583 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:05.583 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:05.583 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:45:05.583 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:45:05.583 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:45:05.583 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:05.583 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.573+0000 7f2d43040640 1 librados: starting msgr at 2024-03-29T11:45:05.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.573+0000 7f2d43040640 1 librados: starting objecter 2024-03-29T11:45:05.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.574+0000 7f2d43040640 1 Processor -- start 2024-03-29T11:45:05.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.574+0000 7f2d43040640 1 -- start start 2024-03-29T11:45:05.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.574+0000 7f2d43040640 1 librados: setting wanted keys 2024-03-29T11:45:05.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.574+0000 7f2d43040640 1 librados: calling monclient init 2024-03-29T11:45:05.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.574+0000 7f2d43040640 10 monclient: init 2024-03-29T11:45:05.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.575+0000 7f2d43040640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:05.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.575+0000 7f2d43040640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:45:05.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.575+0000 7f2d43040640 10 monclient(hunting): picked mon.noname-b con 0x7f2d3c0ad6c0 addr v1:172.21.15.89:6789/0 2024-03-29T11:45:05.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.575+0000 7f2d43040640 10 monclient(hunting): picked mon.noname-a con 0x7f2d3c0af6c0 addr v1:172.21.15.72:6789/0 2024-03-29T11:45:05.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.575+0000 7f2d43040640 10 monclient(hunting): picked mon.noname-c con 0x7f2d3c14fdc0 addr v1:172.21.15.72:6790/0 2024-03-29T11:45:05.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.575+0000 7f2d43040640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d3c0894a0 con 0x7f2d3c0af6c0 2024-03-29T11:45:05.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.575+0000 7f2d43040640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d3c1df860 con 0x7f2d3c0ad6c0 2024-03-29T11:45:05.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.575+0000 7f2d43040640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d3c1517c0 con 0x7f2d3c14fdc0 2024-03-29T11:45:05.584 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.575+0000 7f2d43040640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:05.585 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.575+0000 7f2d43040640 10 monclient(hunting): authenticate will time out at 1464.863386s 2024-03-29T11:45:05.585 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.578+0000 7f2d415dd640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f2d3c14fdc0 0x7f2d3c1f7f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:36948/0 (socket says 172.21.15.72:36948) 2024-03-29T11:45:05.585 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.578+0000 7f2d415dd640 1 -- 172.21.15.72:0/3629370828 learned_addr learned my addr 172.21.15.72:0/3629370828 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:45:05.585 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.578+0000 7f2d397fa640 10 client.?.objecter ms_handle_connect 0x7f2d3c14fdc0 2024-03-29T11:45:05.585 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.578+0000 7f2d397fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:05.588 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.579+0000 7f2d397fa640 1 -- 172.21.15.72:0/3629370828 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2846028888 0 0) 0x7f2d3c1517c0 con 0x7f2d3c14fdc0 2024-03-29T11:45:05.588 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.579+0000 7f2d397fa640 10 monclient(hunting): _negotiate 2024-03-29T11:45:05.588 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.579+0000 7f2d397fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:05.588 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.579+0000 7f2d397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:05.588 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.579+0000 7f2d397fa640 10 monclient(hunting): my global_id is 4604 2024-03-29T11:45:05.588 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.579+0000 7f2d397fa640 1 -- 172.21.15.72:0/3629370828 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d28004eb0 con 0x7f2d3c14fdc0 2024-03-29T11:45:05.588 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.580+0000 7f2d397fa640 1 -- 172.21.15.72:0/3629370828 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2869449105 0 0) 0x7f2d28004eb0 con 0x7f2d3c14fdc0 2024-03-29T11:45:05.588 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.580+0000 7f2d397fa640 1 -- 172.21.15.72:0/3629370828 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d3c1517c0 con 0x7f2d3c14fdc0 2024-03-29T11:45:05.588 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.580+0000 7f2d397fa640 1 -- 172.21.15.72:0/3629370828 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f2d2c001750 con 0x7f2d3c14fdc0 2024-03-29T11:45:05.588 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.580+0000 7f2d397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:05.589 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.580+0000 7f2d397fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:45:05.589 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.580+0000 7f2d397fa640 10 monclient(hunting): dump: 2024-03-29T11:45:05.589 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:45:05.589 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:45:05.589 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:05.589 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:05.589 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:05.589 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:05.589 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:45:05.589 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:45:05.589 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:45:05.589 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:05.589 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.580+0000 7f2d397fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:05.589 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.580+0000 7f2d397fa640 1 -- 172.21.15.72:0/3629370828 <== mon.2 v1:172.21.15.72:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 406979911 0 0) 0x7f2d3c1517c0 con 0x7f2d3c14fdc0 2024-03-29T11:45:05.589 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.580+0000 7f2d397fa640 1 -- 172.21.15.72:0/3629370828 >> v1:172.21.15.89:6789/0 conn(0x7f2d3c0ad6c0 legacy=0x7f2d3c150280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:45:05.590 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.582+0000 7f2d397fa640 1 -- 172.21.15.72:0/3629370828 >> v1:172.21.15.72:6789/0 conn(0x7f2d3c0af6c0 legacy=0x7f2d3c14e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:05.590 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.582+0000 7f2d397fa640 10 monclient: _finish_hunting 0 2024-03-29T11:45:05.590 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.582+0000 7f2d397fa640 1 monclient: found mon.c 2024-03-29T11:45:05.590 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.582+0000 7f2d397fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:05.590 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.582+0000 7f2d397fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:45:05.590 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.582+0000 7f2d397fa640 1 -- 172.21.15.72:0/3629370828 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d3c151950 con 0x7f2d3c14fdc0 2024-03-29T11:45:05.590 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.582+0000 7f2d397fa640 10 monclient: _finish_auth 0 2024-03-29T11:45:05.590 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.582+0000 7f2d397fa640 10 monclient: _check_auth_tickets 2024-03-29T11:45:05.590 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.582+0000 7f2d397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:05.590 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.582+0000 7f2d43040640 5 monclient: authenticate success, global_id 4604 2024-03-29T11:45:05.590 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.582+0000 7f2d43040640 10 monclient: _renew_subs 2024-03-29T11:45:05.590 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.582+0000 7f2d43040640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:45:05.590 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.582+0000 7f2d43040640 1 -- 172.21.15.72:0/3629370828 --> v1:172.21.15.72:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d3c1520c0 con 0x7f2d3c14fdc0 2024-03-29T11:45:05.590 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.582+0000 7f2d43040640 10 client.4604.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:05.590 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.582+0000 7f2d43040640 10 monclient: _renew_subs 2024-03-29T11:45:05.591 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.582+0000 7f2d43040640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:45:05.591 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.582+0000 7f2d43040640 1 -- 172.21.15.72:0/3629370828 --> v1:172.21.15.72:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d3c150b70 con 0x7f2d3c14fdc0 2024-03-29T11:45:05.591 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.582+0000 7f2d43040640 1 librados: init done 2024-03-29T11:45:05.591 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.582+0000 7f2d397fa640 10 client.4604.objecter ms_handle_connect 0x7f2d3c0af6c0 2024-03-29T11:45:05.591 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.582+0000 7f2d397fa640 10 client.4604.objecter resend_mon_ops 2024-03-29T11:45:05.591 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.583+0000 7f2d397fa640 1 -- 172.21.15.72:0/3629370828 <== mon.2 v1:172.21.15.72:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d2c0018b0 con 0x7f2d3c14fdc0 2024-03-29T11:45:05.591 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.584+0000 7f2d397fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:05.591 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.584+0000 7f2d397fa640 1 -- 172.21.15.72:0/3629370828 <== mon.2 v1:172.21.15.72:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f2d2c004a00 con 0x7f2d3c14fdc0 2024-03-29T11:45:05.592 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.584+0000 7f2d3affd640 4 set_mon_vals no callback set 2024-03-29T11:45:05.593 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.585+0000 7f2d397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:05.593 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.585+0000 7f2d397fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:05.593 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.585+0000 7f2d397fa640 10 monclient: dump: 2024-03-29T11:45:05.593 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:45:05.593 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:45:05.593 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:05.593 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:05.593 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:05.593 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:05.593 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:45:05.593 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:45:05.593 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:45:05.593 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:05.593 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.585+0000 7f2d397fa640 1 -- 172.21.15.72:0/3629370828 <== mon.2 v1:172.21.15.72:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f2d2c01a9d0 con 0x7f2d3c14fdc0 2024-03-29T11:45:05.593 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.585+0000 7f2d43040640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:05.594 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.585+0000 7f2d43040640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:05.594 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.585+0000 7f2d43040640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:45:05.594 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.585+0000 7f2d397fa640 1 -- 172.21.15.72:0/3629370828 <== mon.2 v1:172.21.15.72:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3662182158 0 0) 0x7f2d2c0849c0 con 0x7f2d3c14fdc0 2024-03-29T11:45:05.594 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.585+0000 7f2d397fa640 10 client.4604.objecter ms_dispatch 0x7f2d3c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:45:05.594 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.585+0000 7f2d397fa640 3 client.4604.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:45:05.594 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.585+0000 7f2d397fa640 3 client.4604.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:45:05.594 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.586+0000 7f2d397fa640 20 client.4604.objecter dump_active .. 0 homeless 2024-03-29T11:45:05.594 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.586+0000 7f2d43040640 1 -- 172.21.15.72:0/3629370828 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d1c005080 con 0x7f2d3c14fdc0 2024-03-29T11:45:05.594 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.587+0000 7f2d40ddc640 10 monclient: get_auth_request con 0x7f2d2806db80 auth_method 0 2024-03-29T11:45:05.594 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.587+0000 7f2d397fa640 10 client.4604.objecter ms_handle_connect 0x7f2d2806db80 2024-03-29T11:45:05.602 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.598+0000 7f2d397fa640 1 -- 172.21.15.72:0/3629370828 <== mon.2 v1:172.21.15.72:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2d2c0500d0 con 0x7f2d3c14fdc0 2024-03-29T11:45:05.603 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.598+0000 7f2d397fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:05.603 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.599+0000 7f2d397fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:05.929 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.925+0000 7f2d43040640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:05.930 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.925+0000 7f2d43040640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:05.930 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.925+0000 7f2d43040640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:45:05.930 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.925+0000 7f2d43040640 1 -- 172.21.15.72:0/3629370828 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d1c005330 con 0x7f2d3c14fdc0 2024-03-29T11:45:05.930 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.926+0000 7f2d397fa640 1 -- 172.21.15.72:0/3629370828 <== mon.2 v1:172.21.15.72:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3214350691) 0x7f2d2c0521d0 con 0x7f2d3c14fdc0 2024-03-29T11:45:05.930 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.926+0000 7f2d397fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:05.930 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.926+0000 7f2d397fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:05.933 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:45:05.934 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":11,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:45:05.043721+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:45:05.043721+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:05.934 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.930+0000 7f2d43040640 10 librados: watch_flush enter 2024-03-29T11:45:05.934 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.930+0000 7f2d43040640 10 librados: watch_flush exit 2024-03-29T11:45:05.934 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.930+0000 7f2d43040640 20 client.4604.objecter shutdown clearing up homeless session... 2024-03-29T11:45:05.935 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.930+0000 7f2d43040640 10 client.4604.objecter successfully canceled tick 2024-03-29T11:45:05.935 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.930+0000 7f2d43040640 1 -- 172.21.15.72:0/3629370828 >> v1:172.21.15.72:6816/1578866102 conn(0x7f2d2806db80 legacy=0x7f2d28070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:05.935 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.930+0000 7f2d43040640 10 monclient: shutdown 2024-03-29T11:45:05.935 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.930+0000 7f2d43040640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:05.935 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.930+0000 7f2d43040640 1 -- 172.21.15.72:0/3629370828 >> v1:172.21.15.72:6790/0 conn(0x7f2d3c14fdc0 legacy=0x7f2d3c1f7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:05.935 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.930+0000 7f2d43040640 1 -- 172.21.15.72:0/3629370828 shutdown_connections 2024-03-29T11:45:05.935 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.930+0000 7f2d43040640 1 -- 172.21.15.72:0/3629370828 >> 172.21.15.72:0/3629370828 conn(0x7f2d3c0b35e0 msgr2=0x7f2d3c0a9ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:05.936 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.931+0000 7f2d43040640 1 -- 172.21.15.72:0/3629370828 shutdown_connections 2024-03-29T11:45:05.936 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.931+0000 7f2d43040640 1 -- 172.21.15.72:0/3629370828 wait complete. 2024-03-29T11:45:05.936 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:05.932+0000 7f2d43040640 1 librados: shutdown 2024-03-29T11:45:05.954 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:12.762 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:12.914 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.909+0000 7fb00d615640 10 monclient: get_monmap_and_config 2024-03-29T11:45:12.914 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.909+0000 7fb00d615640 10 monclient: build_initial_monmap 2024-03-29T11:45:12.914 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.909+0000 7fb00d615640 10 monclient: monmap: 2024-03-29T11:45:12.914 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:45:12.914 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:45:12.914 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:45:12.910524+0000 2024-03-29T11:45:12.915 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:45:12.910524+0000 2024-03-29T11:45:12.915 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:12.915 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:12.915 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:45:12.915 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:45:12.915 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:45:12.915 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:12.919 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.910+0000 7fb00d615640 1 Processor -- start 2024-03-29T11:45:12.919 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.910+0000 7fb00d615640 1 -- start start 2024-03-29T11:45:12.919 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.910+0000 7fb00d615640 10 monclient: init 2024-03-29T11:45:12.919 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.911+0000 7fb00d615640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:12.919 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.911+0000 7fb00d615640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:45:12.919 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.911+0000 7fb00d615640 10 monclient(hunting): picked mon.noname-a con 0x7fb0081457f0 addr v1:172.21.15.72:6789/0 2024-03-29T11:45:12.919 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.911+0000 7fb00d615640 10 monclient(hunting): picked mon.noname-b con 0x7fb00813f650 addr v1:172.21.15.72:6790/0 2024-03-29T11:45:12.919 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.914+0000 7fb00d615640 10 monclient(hunting): picked mon.noname-c con 0x7fb0080a9d70 addr v1:172.21.15.89:6789/0 2024-03-29T11:45:12.926 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.914+0000 7fb00d615640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0080aaef0 con 0x7fb0081457f0 2024-03-29T11:45:12.927 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.914+0000 7fb0077fe640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7fb0080a9d70 0x7fb00814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:47755/0 (socket says 172.21.15.72:47755) 2024-03-29T11:45:12.927 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.914+0000 7fb0077fe640 1 -- 172.21.15.72:0/1063640592 learned_addr learned my addr 172.21.15.72:0/1063640592 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:45:12.927 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.914+0000 7fb00d615640 1 -- 172.21.15.72:0/1063640592 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb008145bd0 con 0x7fb0080a9d70 2024-03-29T11:45:12.927 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.914+0000 7fb00d615640 1 -- 172.21.15.72:0/1063640592 --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb008145eb0 con 0x7fb00813f650 2024-03-29T11:45:12.927 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.914+0000 7fb00d615640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:12.927 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.914+0000 7fb00d615640 10 monclient(hunting): authenticate will time out at 1472.202968s 2024-03-29T11:45:12.927 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.915+0000 7fb005ffb640 1 -- 172.21.15.72:0/1063640592 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1364705126 0 0) 0x7fb008145bd0 con 0x7fb0080a9d70 2024-03-29T11:45:12.927 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.915+0000 7fb005ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:45:12.927 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.915+0000 7fb005ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:12.927 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.915+0000 7fb005ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:12.928 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.915+0000 7fb005ffb640 10 monclient(hunting): my global_id is 4531 2024-03-29T11:45:12.928 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.915+0000 7fb005ffb640 1 -- 172.21.15.72:0/1063640592 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faff4003520 con 0x7fb0080a9d70 2024-03-29T11:45:12.928 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.915+0000 7fb005ffb640 1 -- 172.21.15.72:0/1063640592 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1515226303 0 0) 0x7faff4003520 con 0x7fb0080a9d70 2024-03-29T11:45:12.928 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.915+0000 7fb005ffb640 1 -- 172.21.15.72:0/1063640592 >> v1:172.21.15.72:6790/0 conn(0x7fb00813f650 legacy=0x7fb0080b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:45:12.928 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.915+0000 7fb005ffb640 1 -- 172.21.15.72:0/1063640592 >> v1:172.21.15.72:6789/0 conn(0x7fb0081457f0 legacy=0x7fb00813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:12.928 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.915+0000 7fb005ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:45:12.928 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.915+0000 7fb005ffb640 1 monclient: found mon.noname-c 2024-03-29T11:45:12.928 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.915+0000 7fb005ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:12.928 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.915+0000 7fb005ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.89:6789/0 2024-03-29T11:45:12.928 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.915+0000 7fb005ffb640 1 -- 172.21.15.72:0/1063640592 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb008146340 con 0x7fb0080a9d70 2024-03-29T11:45:12.929 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.915+0000 7fb005ffb640 10 monclient: _finish_auth 0 2024-03-29T11:45:12.929 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.915+0000 7fb005ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:45:12.929 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.915+0000 7fb005ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:12.929 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.915+0000 7fb005ffb640 1 -- 172.21.15.72:0/1063640592 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fb000001a10 con 0x7fb0080a9d70 2024-03-29T11:45:12.929 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.915+0000 7fb005ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:12.929 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.915+0000 7fb005ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:45:12.929 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.915+0000 7fb005ffb640 10 monclient: dump: 2024-03-29T11:45:12.929 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:45:12.929 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:45:12.929 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:12.930 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:12.930 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:12.930 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:12.930 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:45:12.930 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:45:12.930 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:45:12.930 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:12.930 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.915+0000 7fb00d615640 5 monclient: authenticate success, global_id 4531 2024-03-29T11:45:12.931 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.915+0000 7fb00d615640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:12.931 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.915+0000 7fb005ffb640 1 -- 172.21.15.72:0/1063640592 <== mon.1 v1:172.21.15.89:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb000001be0 con 0x7fb0080a9d70 2024-03-29T11:45:12.931 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.915+0000 7fb005ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:12.931 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.915+0000 7fb005ffb640 1 -- 172.21.15.72:0/1063640592 <== mon.1 v1:172.21.15.89:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fb000002030 con 0x7fb0080a9d70 2024-03-29T11:45:12.931 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.915+0000 7fb00d615640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:12.931 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.915+0000 7fb00d615640 4 set_mon_vals no callback set 2024-03-29T11:45:12.931 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.916+0000 7fb005ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:12.931 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.916+0000 7fb005ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:45:12.931 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.916+0000 7fb005ffb640 10 monclient: dump: 2024-03-29T11:45:12.931 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:45:12.932 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:45:12.932 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:12.932 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:12.932 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:12.933 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:12.934 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:45:12.934 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:45:12.934 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:45:12.934 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:12.934 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.916+0000 7fb00d615640 10 monclient: shutdown 2024-03-29T11:45:12.934 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.916+0000 7fb00d615640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:12.934 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.917+0000 7fb00d615640 1 -- 172.21.15.72:0/1063640592 >> v1:172.21.15.89:6789/0 conn(0x7fb0080a9d70 legacy=0x7fb00814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:12.934 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.919+0000 7fb00d615640 1 -- 172.21.15.72:0/1063640592 shutdown_connections 2024-03-29T11:45:12.934 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.919+0000 7fb00d615640 1 -- 172.21.15.72:0/1063640592 >> 172.21.15.72:0/1063640592 conn(0x7fb008066a00 msgr2=0x7fb008066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:12.934 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.919+0000 7fb00d615640 1 -- 172.21.15.72:0/1063640592 shutdown_connections 2024-03-29T11:45:12.934 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.919+0000 7fb00d615640 1 -- 172.21.15.72:0/1063640592 wait complete. 2024-03-29T11:45:12.934 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.920+0000 7fb00d615640 10 monclient: build_initial_monmap 2024-03-29T11:45:12.934 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.920+0000 7fb00d615640 10 monclient: monmap: 2024-03-29T11:45:12.934 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:45:12.935 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:12.935 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:45:12.935 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:45:12.935 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:12.935 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:12.935 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:45:12.935 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:45:12.935 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:45:12.935 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:12.935 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.921+0000 7fb00d615640 1 librados: starting msgr at 2024-03-29T11:45:12.935 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.921+0000 7fb00d615640 1 librados: starting objecter 2024-03-29T11:45:12.935 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.921+0000 7fb00d615640 1 Processor -- start 2024-03-29T11:45:12.936 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.921+0000 7fb00d615640 1 -- start start 2024-03-29T11:45:12.937 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.921+0000 7fb00d615640 1 librados: setting wanted keys 2024-03-29T11:45:12.937 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.921+0000 7fb00d615640 1 librados: calling monclient init 2024-03-29T11:45:12.937 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.921+0000 7fb00d615640 10 monclient: init 2024-03-29T11:45:12.937 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.922+0000 7fb00d615640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:12.937 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.922+0000 7fb00d615640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:45:12.937 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.923+0000 7fb00d615640 10 monclient(hunting): picked mon.noname-a con 0x7fb0080a9d70 addr v1:172.21.15.72:6789/0 2024-03-29T11:45:12.937 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.926+0000 7fb00d615640 10 monclient(hunting): picked mon.noname-c con 0x7fb00813f650 addr v1:172.21.15.72:6790/0 2024-03-29T11:45:12.937 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.926+0000 7fb00d615640 10 monclient(hunting): picked mon.noname-b con 0x7fb0081457f0 addr v1:172.21.15.89:6789/0 2024-03-29T11:45:12.937 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.926+0000 7fb00d615640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb008142ca0 con 0x7fb0080a9d70 2024-03-29T11:45:12.937 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.926+0000 7fb00d615640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb008142990 con 0x7fb0081457f0 2024-03-29T11:45:12.937 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.926+0000 7fb00d615640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb008141350 con 0x7fb00813f650 2024-03-29T11:45:12.937 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.926+0000 7fb00d615640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:12.937 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.926+0000 7fb00d615640 10 monclient(hunting): authenticate will time out at 1472.214622s 2024-03-29T11:45:12.937 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.929+0000 7fb0067fc640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7fb00813f650 0x7fb008134f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:54669/0 (socket says 172.21.15.72:54669) 2024-03-29T11:45:12.938 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.930+0000 7fb0067fc640 1 -- 172.21.15.72:0/1469402691 learned_addr learned my addr 172.21.15.72:0/1469402691 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:45:12.938 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.930+0000 7fafff7fe640 10 client.?.objecter ms_handle_connect 0x7fb00813f650 2024-03-29T11:45:12.938 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.930+0000 7fafff7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:12.938 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.932+0000 7fafff7fe640 1 -- 172.21.15.72:0/1469402691 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 16857336 0 0) 0x7fb008141350 con 0x7fb00813f650 2024-03-29T11:45:12.938 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.932+0000 7fafff7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:45:12.938 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.932+0000 7fafff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:12.938 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.932+0000 7fafff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:12.938 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.932+0000 7fafff7fe640 10 monclient(hunting): my global_id is 4607 2024-03-29T11:45:12.938 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.932+0000 7fafff7fe640 1 -- 172.21.15.72:0/1469402691 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafe8004eb0 con 0x7fb00813f650 2024-03-29T11:45:12.939 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.933+0000 7fafff7fe640 10 client.?.objecter ms_handle_connect 0x7fb0081457f0 2024-03-29T11:45:12.939 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.933+0000 7fafff7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:12.939 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.933+0000 7fafff7fe640 1 -- 172.21.15.72:0/1469402691 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3691409782 0 0) 0x7fafe8004eb0 con 0x7fb00813f650 2024-03-29T11:45:12.939 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.934+0000 7fafff7fe640 1 -- 172.21.15.72:0/1469402691 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb008141350 con 0x7fb00813f650 2024-03-29T11:45:12.939 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.934+0000 7fafff7fe640 1 -- 172.21.15.72:0/1469402691 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7faff8001960 con 0x7fb00813f650 2024-03-29T11:45:12.939 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.934+0000 7fafff7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:12.939 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.934+0000 7fafff7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:45:12.939 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.934+0000 7fafff7fe640 10 monclient(hunting): dump: 2024-03-29T11:45:12.939 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:45:12.939 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:45:12.939 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:12.940 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:12.940 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:12.940 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:12.940 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:45:12.940 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:45:12.940 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:45:12.940 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:12.940 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.934+0000 7fafff7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:12.942 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.935+0000 7fafff7fe640 10 client.?.objecter ms_handle_connect 0x7fb0080a9d70 2024-03-29T11:45:12.942 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.935+0000 7fafff7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:12.942 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.935+0000 7fafff7fe640 1 -- 172.21.15.72:0/1469402691 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1944278623 0 0) 0x7fb008142ca0 con 0x7fb0080a9d70 2024-03-29T11:45:12.942 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.935+0000 7fafff7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:45:12.943 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.935+0000 7fafff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:12.943 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.935+0000 7fafff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:12.943 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.935+0000 7fafff7fe640 10 monclient(hunting): my global_id is 4581 2024-03-29T11:45:12.943 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.935+0000 7fafff7fe640 1 -- 172.21.15.72:0/1469402691 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafe8004eb0 con 0x7fb0080a9d70 2024-03-29T11:45:12.943 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.935+0000 7fafff7fe640 1 -- 172.21.15.72:0/1469402691 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 494082826 0 0) 0x7fb008142990 con 0x7fb0081457f0 2024-03-29T11:45:12.943 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.935+0000 7fafff7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:45:12.943 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.935+0000 7fafff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:12.943 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.935+0000 7fafff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:12.943 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.935+0000 7fafff7fe640 10 monclient(hunting): my global_id is 4534 2024-03-29T11:45:12.943 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.936+0000 7fafff7fe640 1 -- 172.21.15.72:0/1469402691 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb008142ca0 con 0x7fb0081457f0 2024-03-29T11:45:12.943 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.936+0000 7fafff7fe640 1 -- 172.21.15.72:0/1469402691 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2733124252 0 0) 0x7fafe8004eb0 con 0x7fb0080a9d70 2024-03-29T11:45:12.943 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.936+0000 7fafff7fe640 1 -- 172.21.15.72:0/1469402691 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb008142990 con 0x7fb0080a9d70 2024-03-29T11:45:12.943 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.936+0000 7fafff7fe640 1 -- 172.21.15.72:0/1469402691 <== mon.2 v1:172.21.15.72:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3992924941 0 0) 0x7fb008141350 con 0x7fb00813f650 2024-03-29T11:45:12.943 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.936+0000 7fafff7fe640 1 -- 172.21.15.72:0/1469402691 >> v1:172.21.15.89:6789/0 conn(0x7fb0081457f0 legacy=0x7fb0081d7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:12.943 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.936+0000 7fafff7fe640 1 -- 172.21.15.72:0/1469402691 >> v1:172.21.15.72:6789/0 conn(0x7fb0080a9d70 legacy=0x7fb008140c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:12.944 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.936+0000 7fafff7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:45:12.944 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.936+0000 7fafff7fe640 1 monclient: found mon.c 2024-03-29T11:45:12.944 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.936+0000 7fafff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:12.944 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.936+0000 7fafff7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:45:12.944 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.936+0000 7fafff7fe640 1 -- 172.21.15.72:0/1469402691 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0081414e0 con 0x7fb00813f650 2024-03-29T11:45:12.944 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.936+0000 7fafff7fe640 10 monclient: _finish_auth 0 2024-03-29T11:45:12.944 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.937+0000 7fafff7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:45:12.944 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.937+0000 7fafff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:12.944 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.939+0000 7fb00d615640 5 monclient: authenticate success, global_id 4607 2024-03-29T11:45:12.944 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.939+0000 7fb00d615640 10 monclient: _renew_subs 2024-03-29T11:45:12.944 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.939+0000 7fb00d615640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:45:12.945 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.939+0000 7fb00d615640 1 -- 172.21.15.72:0/1469402691 --> v1:172.21.15.72:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb008135650 con 0x7fb00813f650 2024-03-29T11:45:12.945 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.939+0000 7fb00d615640 10 client.4607.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:12.945 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.939+0000 7fb00d615640 10 monclient: _renew_subs 2024-03-29T11:45:12.945 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.939+0000 7fb00d615640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:45:12.945 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.939+0000 7fb00d615640 1 -- 172.21.15.72:0/1469402691 --> v1:172.21.15.72:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb008140110 con 0x7fb00813f650 2024-03-29T11:45:12.945 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.939+0000 7fb00d615640 1 librados: init done 2024-03-29T11:45:12.945 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.940+0000 7fafff7fe640 1 -- 172.21.15.72:0/1469402691 <== mon.2 v1:172.21.15.72:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faff8001d00 con 0x7fb00813f650 2024-03-29T11:45:12.945 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.940+0000 7fafff7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:12.945 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.940+0000 7fafff7fe640 1 -- 172.21.15.72:0/1469402691 <== mon.2 v1:172.21.15.72:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7faff80038a0 con 0x7fb00813f650 2024-03-29T11:45:12.945 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.940+0000 7fafff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:12.945 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.940+0000 7fafff7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:12.945 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.940+0000 7fafff7fe640 10 monclient: dump: 2024-03-29T11:45:12.945 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:45:12.945 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:45:12.945 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:12.945 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:12.946 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:12.946 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:12.946 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:45:12.946 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:45:12.946 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:45:12.946 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:12.947 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.940+0000 7fb0057fa640 4 set_mon_vals no callback set 2024-03-29T11:45:12.947 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.941+0000 7fb00d615640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:12.947 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.941+0000 7fb00d615640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:12.947 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.941+0000 7fb00d615640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:45:12.947 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.941+0000 7fb00d615640 1 -- 172.21.15.72:0/1469402691 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafec005080 con 0x7fb00813f650 2024-03-29T11:45:12.947 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.942+0000 7fafff7fe640 1 -- 172.21.15.72:0/1469402691 <== mon.2 v1:172.21.15.72:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7faff8003a00 con 0x7fb00813f650 2024-03-29T11:45:12.947 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.942+0000 7fafff7fe640 1 -- 172.21.15.72:0/1469402691 <== mon.2 v1:172.21.15.72:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3662182158 0 0) 0x7faff8084a20 con 0x7fb00813f650 2024-03-29T11:45:12.948 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.943+0000 7fafff7fe640 10 client.4607.objecter ms_dispatch 0x7fb008000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:45:12.948 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.943+0000 7fafff7fe640 3 client.4607.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:45:12.948 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.943+0000 7fafff7fe640 3 client.4607.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:45:12.948 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.943+0000 7fafff7fe640 20 client.4607.objecter dump_active .. 0 homeless 2024-03-29T11:45:12.948 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.943+0000 7fb006ffd640 10 monclient: get_auth_request con 0x7fafe806ddf0 auth_method 0 2024-03-29T11:45:12.952 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.948+0000 7fafff7fe640 10 client.4607.objecter ms_handle_connect 0x7fafe806ddf0 2024-03-29T11:45:12.957 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.953+0000 7fafff7fe640 1 -- 172.21.15.72:0/1469402691 <== mon.2 v1:172.21.15.72:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faff8050130 con 0x7fb00813f650 2024-03-29T11:45:12.957 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.953+0000 7fafff7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:12.957 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:12.953+0000 7fafff7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:13.286 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:13.280+0000 7fb00d615640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:13.286 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:13.280+0000 7fb00d615640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:13.286 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:13.281+0000 7fb00d615640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:45:13.286 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:13.281+0000 7fb00d615640 1 -- 172.21.15.72:0/1469402691 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fafec005330 con 0x7fb00813f650 2024-03-29T11:45:13.289 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:13.284+0000 7fafff7fe640 1 -- 172.21.15.72:0/1469402691 <== mon.2 v1:172.21.15.72:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3214350691) 0x7faff8052230 con 0x7fb00813f650 2024-03-29T11:45:13.289 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:13.284+0000 7fafff7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:13.289 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:13.284+0000 7fafff7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:13.290 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:45:13.290 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":11,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:45:05.043721+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:45:05.043721+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:13.298 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:13.289+0000 7fb00d615640 10 librados: watch_flush enter 2024-03-29T11:45:13.298 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:13.289+0000 7fb00d615640 10 librados: watch_flush exit 2024-03-29T11:45:13.298 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:13.289+0000 7fb00d615640 20 client.4607.objecter shutdown clearing up homeless session... 2024-03-29T11:45:13.298 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:13.289+0000 7fb00d615640 10 client.4607.objecter successfully canceled tick 2024-03-29T11:45:13.298 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:13.289+0000 7fb00d615640 1 -- 172.21.15.72:0/1469402691 >> v1:172.21.15.72:6816/1578866102 conn(0x7fafe806ddf0 legacy=0x7fafe80702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:13.298 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:13.289+0000 7fb00d615640 10 monclient: shutdown 2024-03-29T11:45:13.298 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:13.289+0000 7fb00d615640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:13.299 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:13.289+0000 7fb00d615640 1 -- 172.21.15.72:0/1469402691 >> v1:172.21.15.72:6790/0 conn(0x7fb00813f650 legacy=0x7fb008134f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:13.299 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:13.290+0000 7fb00d615640 1 -- 172.21.15.72:0/1469402691 shutdown_connections 2024-03-29T11:45:13.299 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:13.290+0000 7fb00d615640 1 -- 172.21.15.72:0/1469402691 >> 172.21.15.72:0/1469402691 conn(0x7fb008066a00 msgr2=0x7fb0080b4360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:13.299 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:13.290+0000 7fb00d615640 1 -- 172.21.15.72:0/1469402691 shutdown_connections 2024-03-29T11:45:13.299 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:13.291+0000 7fb00d615640 1 -- 172.21.15.72:0/1469402691 wait complete. 2024-03-29T11:45:13.299 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:13.291+0000 7fb00d615640 1 librados: shutdown 2024-03-29T11:45:13.314 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:20.122 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:20.271 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.266+0000 7fb27a11c640 10 monclient: get_monmap_and_config 2024-03-29T11:45:20.272 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.266+0000 7fb27a11c640 10 monclient: build_initial_monmap 2024-03-29T11:45:20.272 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.266+0000 7fb27a11c640 10 monclient: monmap: 2024-03-29T11:45:20.272 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:45:20.272 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:45:20.272 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:45:20.267900+0000 2024-03-29T11:45:20.272 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:45:20.267900+0000 2024-03-29T11:45:20.272 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:20.272 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:20.272 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:45:20.272 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:45:20.272 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:45:20.272 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:20.273 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.268+0000 7fb27a11c640 1 Processor -- start 2024-03-29T11:45:20.273 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.268+0000 7fb27a11c640 1 -- start start 2024-03-29T11:45:20.273 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.268+0000 7fb27a11c640 10 monclient: init 2024-03-29T11:45:20.276 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.269+0000 7fb27a11c640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:20.276 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.269+0000 7fb27a11c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:45:20.276 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.269+0000 7fb27a11c640 10 monclient(hunting): picked mon.noname-b con 0x7fb2740af0f0 addr v1:172.21.15.72:6790/0 2024-03-29T11:45:20.277 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.269+0000 7fb27a11c640 10 monclient(hunting): picked mon.noname-a con 0x7fb2740af4d0 addr v1:172.21.15.72:6789/0 2024-03-29T11:45:20.277 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.269+0000 7fb27a11c640 10 monclient(hunting): picked mon.noname-c con 0x7fb2740ab170 addr v1:172.21.15.89:6789/0 2024-03-29T11:45:20.277 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.269+0000 7fb27a11c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb27413f010 con 0x7fb2740af4d0 2024-03-29T11:45:20.277 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.269+0000 7fb27a11c640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2740ab6c0 con 0x7fb2740ab170 2024-03-29T11:45:20.277 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.269+0000 7fb27a11c640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2740ab9a0 con 0x7fb2740af0f0 2024-03-29T11:45:20.277 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.269+0000 7fb27a11c640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:20.277 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.269+0000 7fb27a11c640 10 monclient(hunting): authenticate will time out at 1479.557641s 2024-03-29T11:45:20.277 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.272+0000 7fb2737fe640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7fb2740af0f0 0x7fb27413c3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:40966/0 (socket says 172.21.15.72:40966) 2024-03-29T11:45:20.277 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.272+0000 7fb2737fe640 1 -- 172.21.15.72:0/3465505041 learned_addr learned my addr 172.21.15.72:0/3465505041 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:45:20.278 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.273+0000 7fb2727fc640 1 -- 172.21.15.72:0/3465505041 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1924416592 0 0) 0x7fb2740ab9a0 con 0x7fb2740af0f0 2024-03-29T11:45:20.278 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.273+0000 7fb2727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:45:20.278 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.273+0000 7fb2727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:20.278 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.273+0000 7fb2727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:20.278 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.273+0000 7fb2727fc640 10 monclient(hunting): my global_id is 4610 2024-03-29T11:45:20.278 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.273+0000 7fb2727fc640 1 -- 172.21.15.72:0/3465505041 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb25c003520 con 0x7fb2740af0f0 2024-03-29T11:45:20.278 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.273+0000 7fb2727fc640 1 -- 172.21.15.72:0/3465505041 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3180768510 0 0) 0x7fb2740ab6c0 con 0x7fb2740ab170 2024-03-29T11:45:20.278 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.273+0000 7fb2727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:45:20.278 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.273+0000 7fb2727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:20.278 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.273+0000 7fb2727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:20.279 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.273+0000 7fb2727fc640 10 monclient(hunting): my global_id is 4537 2024-03-29T11:45:20.279 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.273+0000 7fb2727fc640 1 -- 172.21.15.72:0/3465505041 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2740ab9a0 con 0x7fb2740ab170 2024-03-29T11:45:20.279 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.273+0000 7fb2727fc640 1 -- 172.21.15.72:0/3465505041 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1649931101 0 0) 0x7fb27413f010 con 0x7fb2740af4d0 2024-03-29T11:45:20.279 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.273+0000 7fb2727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:45:20.279 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.273+0000 7fb2727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:20.279 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.273+0000 7fb2727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:20.279 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.273+0000 7fb2727fc640 10 monclient(hunting): my global_id is 4584 2024-03-29T11:45:20.279 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.273+0000 7fb2727fc640 1 -- 172.21.15.72:0/3465505041 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2740ab6c0 con 0x7fb2740af4d0 2024-03-29T11:45:20.279 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.273+0000 7fb2727fc640 1 -- 172.21.15.72:0/3465505041 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 297934104 0 0) 0x7fb2740ab9a0 con 0x7fb2740ab170 2024-03-29T11:45:20.279 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.273+0000 7fb2727fc640 1 -- 172.21.15.72:0/3465505041 >> v1:172.21.15.72:6790/0 conn(0x7fb2740af0f0 legacy=0x7fb27413c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:20.279 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.273+0000 7fb2727fc640 1 -- 172.21.15.72:0/3465505041 >> v1:172.21.15.72:6789/0 conn(0x7fb2740af4d0 legacy=0x7fb27413f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:20.279 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.273+0000 7fb2727fc640 10 monclient: _finish_hunting 0 2024-03-29T11:45:20.279 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.273+0000 7fb2727fc640 1 monclient: found mon.noname-c 2024-03-29T11:45:20.279 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.273+0000 7fb2727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:20.279 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.273+0000 7fb2727fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.89:6789/0 2024-03-29T11:45:20.280 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.273+0000 7fb2727fc640 1 -- 172.21.15.72:0/3465505041 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2740abc80 con 0x7fb2740ab170 2024-03-29T11:45:20.280 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.273+0000 7fb2727fc640 10 monclient: _finish_auth 0 2024-03-29T11:45:20.280 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.273+0000 7fb2727fc640 10 monclient: _check_auth_tickets 2024-03-29T11:45:20.280 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.273+0000 7fb2727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:20.280 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.273+0000 7fb2727fc640 1 -- 172.21.15.72:0/3465505041 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fb26c0019f0 con 0x7fb2740ab170 2024-03-29T11:45:20.280 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.273+0000 7fb2727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:20.280 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.273+0000 7fb2727fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:45:20.280 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.273+0000 7fb2727fc640 10 monclient: dump: 2024-03-29T11:45:20.280 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:45:20.280 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:45:20.280 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:20.280 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:20.280 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:20.280 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:20.280 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:45:20.280 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:45:20.281 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:45:20.281 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:20.281 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.273+0000 7fb27a11c640 5 monclient: authenticate success, global_id 4537 2024-03-29T11:45:20.281 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.273+0000 7fb27a11c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:20.282 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.274+0000 7fb2727fc640 1 -- 172.21.15.72:0/3465505041 <== mon.1 v1:172.21.15.89:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb26c001c10 con 0x7fb2740ab170 2024-03-29T11:45:20.282 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.274+0000 7fb2727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:20.282 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.274+0000 7fb2727fc640 1 -- 172.21.15.72:0/3465505041 <== mon.1 v1:172.21.15.89:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fb26c001fa0 con 0x7fb2740ab170 2024-03-29T11:45:20.282 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.274+0000 7fb2727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:20.282 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.274+0000 7fb2727fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:45:20.282 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.274+0000 7fb2727fc640 10 monclient: dump: 2024-03-29T11:45:20.282 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:45:20.282 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:45:20.282 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:20.282 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:20.282 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:20.282 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:20.282 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:45:20.283 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:45:20.283 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:45:20.283 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:20.283 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.274+0000 7fb27a11c640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:20.283 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.274+0000 7fb27a11c640 4 set_mon_vals no callback set 2024-03-29T11:45:20.283 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.274+0000 7fb27a11c640 10 monclient: shutdown 2024-03-29T11:45:20.283 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.274+0000 7fb27a11c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:20.283 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.274+0000 7fb27a11c640 1 -- 172.21.15.72:0/3465505041 >> v1:172.21.15.89:6789/0 conn(0x7fb2740ab170 legacy=0x7fb27414a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:20.283 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.275+0000 7fb27a11c640 1 -- 172.21.15.72:0/3465505041 shutdown_connections 2024-03-29T11:45:20.283 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.275+0000 7fb27a11c640 1 -- 172.21.15.72:0/3465505041 >> 172.21.15.72:0/3465505041 conn(0x7fb2740b35e0 msgr2=0x7fb2740b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:20.283 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.275+0000 7fb27a11c640 1 -- 172.21.15.72:0/3465505041 shutdown_connections 2024-03-29T11:45:20.283 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.275+0000 7fb27a11c640 1 -- 172.21.15.72:0/3465505041 wait complete. 2024-03-29T11:45:20.283 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.275+0000 7fb27a11c640 10 monclient: build_initial_monmap 2024-03-29T11:45:20.283 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.275+0000 7fb27a11c640 10 monclient: monmap: 2024-03-29T11:45:20.283 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:45:20.283 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:20.283 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:45:20.284 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:45:20.286 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:20.286 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:20.286 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:45:20.286 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:45:20.286 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:45:20.286 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:20.286 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.276+0000 7fb27a11c640 1 librados: starting msgr at 2024-03-29T11:45:20.286 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.276+0000 7fb27a11c640 1 librados: starting objecter 2024-03-29T11:45:20.286 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.276+0000 7fb27a11c640 1 Processor -- start 2024-03-29T11:45:20.289 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.277+0000 7fb27a11c640 1 -- start start 2024-03-29T11:45:20.289 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.277+0000 7fb27a11c640 1 librados: setting wanted keys 2024-03-29T11:45:20.289 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.277+0000 7fb27a11c640 1 librados: calling monclient init 2024-03-29T11:45:20.289 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.277+0000 7fb27a11c640 10 monclient: init 2024-03-29T11:45:20.289 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.278+0000 7fb27a11c640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:20.289 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.278+0000 7fb27a11c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:45:20.289 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.278+0000 7fb27a11c640 10 monclient(hunting): picked mon.noname-a con 0x7fb2740ab170 addr v1:172.21.15.72:6789/0 2024-03-29T11:45:20.289 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.278+0000 7fb27a11c640 10 monclient(hunting): picked mon.noname-c con 0x7fb2740af0f0 addr v1:172.21.15.72:6790/0 2024-03-29T11:45:20.289 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.278+0000 7fb27a11c640 10 monclient(hunting): picked mon.noname-b con 0x7fb2741c82a0 addr v1:172.21.15.89:6789/0 2024-03-29T11:45:20.290 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.278+0000 7fb27a11c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb27406efb0 con 0x7fb2740ab170 2024-03-29T11:45:20.290 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.278+0000 7fb27a11c640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2741de6a0 con 0x7fb2741c82a0 2024-03-29T11:45:20.290 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.278+0000 7fb27a11c640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb27406f220 con 0x7fb2740af0f0 2024-03-29T11:45:20.290 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.278+0000 7fb27a11c640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:20.290 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.278+0000 7fb27a11c640 10 monclient(hunting): authenticate will time out at 1479.566187s 2024-03-29T11:45:20.290 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.280+0000 7fb2737fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fb2740ab170 0x7fb2741de970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:59131/0 (socket says 172.21.15.72:59131) 2024-03-29T11:45:20.290 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.280+0000 7fb2737fe640 1 -- 172.21.15.72:0/3734983033 learned_addr learned my addr 172.21.15.72:0/3734983033 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:45:20.290 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.280+0000 7fb25bfff640 10 client.?.objecter ms_handle_connect 0x7fb2740ab170 2024-03-29T11:45:20.290 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.280+0000 7fb25bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:20.290 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.281+0000 7fb25bfff640 10 client.?.objecter ms_handle_connect 0x7fb2740af0f0 2024-03-29T11:45:20.290 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.281+0000 7fb25bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:20.290 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.281+0000 7fb25bfff640 1 -- 172.21.15.72:0/3734983033 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 209684926 0 0) 0x7fb27406efb0 con 0x7fb2740ab170 2024-03-29T11:45:20.290 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.281+0000 7fb25bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:45:20.290 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.281+0000 7fb25bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:20.291 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.281+0000 7fb25bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:20.291 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.281+0000 7fb25bfff640 10 monclient(hunting): my global_id is 4587 2024-03-29T11:45:20.291 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.281+0000 7fb25bfff640 1 -- 172.21.15.72:0/3734983033 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb260004eb0 con 0x7fb2740ab170 2024-03-29T11:45:20.291 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.281+0000 7fb25bfff640 1 -- 172.21.15.72:0/3734983033 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 304706370 0 0) 0x7fb27406f220 con 0x7fb2740af0f0 2024-03-29T11:45:20.291 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.281+0000 7fb25bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:45:20.291 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.281+0000 7fb25bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:20.291 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.281+0000 7fb25bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:20.291 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.281+0000 7fb25bfff640 10 monclient(hunting): my global_id is 4613 2024-03-29T11:45:20.291 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.281+0000 7fb25bfff640 1 -- 172.21.15.72:0/3734983033 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb27406efb0 con 0x7fb2740af0f0 2024-03-29T11:45:20.291 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.282+0000 7fb25bfff640 1 -- 172.21.15.72:0/3734983033 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 551751132 0 0) 0x7fb260004eb0 con 0x7fb2740ab170 2024-03-29T11:45:20.291 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.282+0000 7fb25bfff640 1 -- 172.21.15.72:0/3734983033 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb27406f220 con 0x7fb2740ab170 2024-03-29T11:45:20.291 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.282+0000 7fb25bfff640 1 -- 172.21.15.72:0/3734983033 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fb268001950 con 0x7fb2740ab170 2024-03-29T11:45:20.291 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.282+0000 7fb25bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:20.291 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.282+0000 7fb25bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:45:20.291 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.282+0000 7fb25bfff640 10 monclient(hunting): dump: 2024-03-29T11:45:20.291 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:45:20.292 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:45:20.292 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:20.292 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:20.292 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:20.292 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:20.292 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:45:20.292 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:45:20.292 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:45:20.292 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:20.292 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.282+0000 7fb25bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:20.292 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.282+0000 7fb25bfff640 1 -- 172.21.15.72:0/3734983033 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 622573183 0 0) 0x7fb27406efb0 con 0x7fb2740af0f0 2024-03-29T11:45:20.292 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.282+0000 7fb25bfff640 1 -- 172.21.15.72:0/3734983033 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb260004eb0 con 0x7fb2740af0f0 2024-03-29T11:45:20.293 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.282+0000 7fb25bfff640 1 -- 172.21.15.72:0/3734983033 <== mon.0 v1:172.21.15.72:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3405711254 0 0) 0x7fb27406f220 con 0x7fb2740ab170 2024-03-29T11:45:20.293 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.282+0000 7fb25bfff640 1 -- 172.21.15.72:0/3734983033 >> v1:172.21.15.72:6790/0 conn(0x7fb2740af0f0 legacy=0x7fb2741dd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:20.293 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.282+0000 7fb25bfff640 1 -- 172.21.15.72:0/3734983033 >> v1:172.21.15.89:6789/0 conn(0x7fb2741c82a0 legacy=0x7fb2741ddf90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:45:20.293 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.282+0000 7fb25bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:45:20.293 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.282+0000 7fb25bfff640 1 monclient: found mon.a 2024-03-29T11:45:20.293 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.282+0000 7fb25bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:20.293 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.282+0000 7fb25bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:45:20.293 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.282+0000 7fb25bfff640 1 -- 172.21.15.72:0/3734983033 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb27406f500 con 0x7fb2740ab170 2024-03-29T11:45:20.293 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.282+0000 7fb25bfff640 10 monclient: _finish_auth 0 2024-03-29T11:45:20.293 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.282+0000 7fb25bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:45:20.293 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.282+0000 7fb25bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:20.293 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.282+0000 7fb27a11c640 5 monclient: authenticate success, global_id 4587 2024-03-29T11:45:20.293 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.282+0000 7fb27a11c640 10 monclient: _renew_subs 2024-03-29T11:45:20.293 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.282+0000 7fb27a11c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:45:20.293 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.282+0000 7fb27a11c640 1 -- 172.21.15.72:0/3734983033 --> v1:172.21.15.72:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb27413e640 con 0x7fb2740ab170 2024-03-29T11:45:20.294 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.283+0000 7fb27a11c640 10 client.4587.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:20.294 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.283+0000 7fb27a11c640 10 monclient: _renew_subs 2024-03-29T11:45:20.294 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.283+0000 7fb27a11c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:45:20.294 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.283+0000 7fb27a11c640 1 -- 172.21.15.72:0/3734983033 --> v1:172.21.15.72:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2741c8e00 con 0x7fb2740ab170 2024-03-29T11:45:20.294 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.283+0000 7fb27a11c640 1 librados: init done 2024-03-29T11:45:20.294 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.284+0000 7fb25bfff640 1 -- 172.21.15.72:0/3734983033 <== mon.0 v1:172.21.15.72:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb2680047e0 con 0x7fb2740ab170 2024-03-29T11:45:20.294 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.284+0000 7fb25bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:20.294 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.284+0000 7fb25bfff640 1 -- 172.21.15.72:0/3734983033 <== mon.0 v1:172.21.15.72:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7fb268001d80 con 0x7fb2740ab170 2024-03-29T11:45:20.294 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.284+0000 7fb25bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:20.294 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.284+0000 7fb25bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:45:20.294 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.284+0000 7fb25bfff640 10 monclient: dump: 2024-03-29T11:45:20.294 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:45:20.294 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:45:20.294 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:20.294 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:20.295 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:20.295 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:20.295 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:45:20.295 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:45:20.295 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:45:20.295 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:20.295 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.284+0000 7fb259ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:20.295 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.284+0000 7fb259ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:20.295 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.284+0000 7fb259ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:45:20.295 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.284+0000 7fb259ffb640 1 -- 172.21.15.72:0/3734983033 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb254005080 con 0x7fb2740ab170 2024-03-29T11:45:20.296 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.284+0000 7fb271ffb640 4 set_mon_vals no callback set 2024-03-29T11:45:20.296 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.285+0000 7fb25bfff640 1 -- 172.21.15.72:0/3734983033 <== mon.0 v1:172.21.15.72:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7fb268004b70 con 0x7fb2740ab170 2024-03-29T11:45:20.296 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.286+0000 7fb272ffd640 10 monclient: get_auth_request con 0x7fb26006df00 auth_method 0 2024-03-29T11:45:20.296 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.286+0000 7fb25bfff640 1 -- 172.21.15.72:0/3734983033 <== mon.0 v1:172.21.15.72:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3662182158 0 0) 0x7fb268084890 con 0x7fb2740ab170 2024-03-29T11:45:20.296 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.286+0000 7fb25bfff640 10 client.4587.objecter ms_dispatch 0x7fb274000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:45:20.297 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.286+0000 7fb25bfff640 3 client.4587.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:45:20.297 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.286+0000 7fb25bfff640 3 client.4587.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:45:20.297 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.286+0000 7fb25bfff640 20 client.4587.objecter dump_active .. 0 homeless 2024-03-29T11:45:20.297 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.286+0000 7fb25bfff640 10 client.4587.objecter ms_handle_connect 0x7fb26006df00 2024-03-29T11:45:20.302 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.294+0000 7fb25bfff640 1 -- 172.21.15.72:0/3734983033 <== mon.0 v1:172.21.15.72:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb2680501d0 con 0x7fb2740ab170 2024-03-29T11:45:20.302 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.295+0000 7fb25bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:20.302 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.295+0000 7fb25bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:20.636 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.632+0000 7fb259ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:20.636 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.632+0000 7fb259ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:20.636 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.632+0000 7fb259ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:45:20.636 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.632+0000 7fb259ffb640 1 -- 172.21.15.72:0/3734983033 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb254005330 con 0x7fb2740ab170 2024-03-29T11:45:20.637 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.633+0000 7fb25bfff640 1 -- 172.21.15.72:0/3734983033 <== mon.0 v1:172.21.15.72:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3214350691) 0x7fb2680049d0 con 0x7fb2740ab170 2024-03-29T11:45:20.637 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.633+0000 7fb25bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:20.638 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.633+0000 7fb25bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:20.638 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:45:20.638 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":11,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:45:05.043721+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:45:05.043721+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:20.641 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.637+0000 7fb259ffb640 10 librados: watch_flush enter 2024-03-29T11:45:20.642 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.638+0000 7fb259ffb640 10 librados: watch_flush exit 2024-03-29T11:45:20.642 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.638+0000 7fb259ffb640 20 client.4587.objecter shutdown clearing up homeless session... 2024-03-29T11:45:20.642 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.638+0000 7fb259ffb640 10 client.4587.objecter successfully canceled tick 2024-03-29T11:45:20.643 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.638+0000 7fb259ffb640 1 -- 172.21.15.72:0/3734983033 >> v1:172.21.15.72:6816/1578866102 conn(0x7fb26006df00 legacy=0x7fb2600703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:20.643 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.638+0000 7fb259ffb640 10 monclient: shutdown 2024-03-29T11:45:20.643 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.638+0000 7fb259ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:20.643 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.638+0000 7fb259ffb640 1 -- 172.21.15.72:0/3734983033 >> v1:172.21.15.72:6789/0 conn(0x7fb2740ab170 legacy=0x7fb2741de970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:20.643 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.638+0000 7fb259ffb640 1 -- 172.21.15.72:0/3734983033 shutdown_connections 2024-03-29T11:45:20.643 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.638+0000 7fb259ffb640 1 -- 172.21.15.72:0/3734983033 >> 172.21.15.72:0/3734983033 conn(0x7fb2740b35e0 msgr2=0x7fb274139840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:20.644 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.639+0000 7fb259ffb640 1 -- 172.21.15.72:0/3734983033 shutdown_connections 2024-03-29T11:45:20.644 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.640+0000 7fb259ffb640 1 -- 172.21.15.72:0/3734983033 wait complete. 2024-03-29T11:45:20.648 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:20.644+0000 7fb259ffb640 1 librados: shutdown 2024-03-29T11:45:20.666 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:27.473 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:27.734 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.619+0000 7f167b59e640 10 monclient: get_monmap_and_config 2024-03-29T11:45:27.734 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.619+0000 7f167b59e640 10 monclient: build_initial_monmap 2024-03-29T11:45:27.734 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.619+0000 7f167b59e640 10 monclient: monmap: 2024-03-29T11:45:27.734 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:45:27.734 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:45:27.735 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:45:27.620153+0000 2024-03-29T11:45:27.735 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:45:27.620153+0000 2024-03-29T11:45:27.735 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:27.735 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:27.735 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:45:27.735 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:45:27.735 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:45:27.735 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:27.735 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.620+0000 7f167b59e640 1 Processor -- start 2024-03-29T11:45:27.735 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.620+0000 7f167b59e640 1 -- start start 2024-03-29T11:45:27.735 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.620+0000 7f167b59e640 10 monclient: init 2024-03-29T11:45:27.735 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.621+0000 7f167b59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:27.735 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.621+0000 7f167b59e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:45:27.735 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.622+0000 7f167b59e640 10 monclient(hunting): picked mon.noname-b con 0x7f167c0a9750 addr v1:172.21.15.72:6790/0 2024-03-29T11:45:27.735 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.622+0000 7f167b59e640 10 monclient(hunting): picked mon.noname-a con 0x7f167c0ae930 addr v1:172.21.15.72:6789/0 2024-03-29T11:45:27.736 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.622+0000 7f167b59e640 10 monclient(hunting): picked mon.noname-c con 0x7f167c0acab0 addr v1:172.21.15.89:6789/0 2024-03-29T11:45:27.736 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.622+0000 7f167b59e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f167c0ad650 con 0x7f167c0ae930 2024-03-29T11:45:27.736 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.622+0000 7f167b59e640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f167c0aedc0 con 0x7f167c0acab0 2024-03-29T11:45:27.736 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.622+0000 7f167b59e640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f167c0ab750 con 0x7f167c0a9750 2024-03-29T11:45:27.736 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.622+0000 7f167b59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:27.736 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.622+0000 7f167b59e640 10 monclient(hunting): authenticate will time out at 1486.910257s 2024-03-29T11:45:27.736 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.626+0000 7f1679d9b640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f167c0ae930 0x7f167c0ac3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:38833/0 (socket says 172.21.15.72:38833) 2024-03-29T11:45:27.736 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.626+0000 7f1679d9b640 1 -- 172.21.15.72:0/1463077548 learned_addr learned my addr 172.21.15.72:0/1463077548 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:45:27.736 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.627+0000 7f167959a640 1 -- 172.21.15.72:0/1463077548 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1136342872 0 0) 0x7f1674001490 con 0x7f167c0ae930 2024-03-29T11:45:27.736 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.627+0000 7f167959a640 10 monclient(hunting): _negotiate 2024-03-29T11:45:27.736 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.627+0000 7f167959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:27.736 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.627+0000 7f167959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:27.736 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.627+0000 7f167959a640 10 monclient(hunting): my global_id is 4590 2024-03-29T11:45:27.736 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.627+0000 7f167959a640 1 -- 172.21.15.72:0/1463077548 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1664003520 con 0x7f167c0ae930 2024-03-29T11:45:27.736 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.627+0000 7f167959a640 1 -- 172.21.15.72:0/1463077548 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1920689764 0 0) 0x7f1664003520 con 0x7f167c0ae930 2024-03-29T11:45:27.736 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.729+0000 7f167959a640 1 -- 172.21.15.72:0/1463077548 >> v1:172.21.15.72:6790/0 conn(0x7f167c0a9750 legacy=0x7f167c0ae220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:27.737 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.729+0000 7f167959a640 1 -- 172.21.15.72:0/1463077548 >> v1:172.21.15.89:6789/0 conn(0x7f167c0acab0 legacy=0x7f167c0acf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:27.738 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.729+0000 7f167959a640 10 monclient: _finish_hunting 0 2024-03-29T11:45:27.738 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.729+0000 7f167959a640 1 monclient: found mon.noname-a 2024-03-29T11:45:27.738 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.729+0000 7f167959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:27.738 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.729+0000 7f167959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.72:6789/0 2024-03-29T11:45:27.738 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.729+0000 7f167959a640 1 -- 172.21.15.72:0/1463077548 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f167c0ab9c0 con 0x7f167c0ae930 2024-03-29T11:45:27.738 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.729+0000 7f167959a640 10 monclient: _finish_auth 0 2024-03-29T11:45:27.738 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.729+0000 7f167959a640 10 monclient: _check_auth_tickets 2024-03-29T11:45:27.738 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.729+0000 7f167959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:27.739 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.729+0000 7f167959a640 1 -- 172.21.15.72:0/1463077548 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f1674001a00 con 0x7f167c0ae930 2024-03-29T11:45:27.739 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.729+0000 7f167b59e640 5 monclient: authenticate success, global_id 4590 2024-03-29T11:45:27.739 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.729+0000 7f167b59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:27.739 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.729+0000 7f167959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:27.739 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.729+0000 7f167959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:45:27.739 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.729+0000 7f167959a640 10 monclient: dump: 2024-03-29T11:45:27.739 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:45:27.739 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:45:27.739 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:27.739 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:27.739 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:27.739 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:27.740 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:45:27.740 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:45:27.740 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:45:27.740 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:27.740 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.729+0000 7f167b59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:27.740 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.732+0000 7f167959a640 1 -- 172.21.15.72:0/1463077548 <== mon.0 v1:172.21.15.72:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1674001c90 con 0x7f167c0ae930 2024-03-29T11:45:27.740 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.732+0000 7f167959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:27.741 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.732+0000 7f167959a640 1 -- 172.21.15.72:0/1463077548 <== mon.0 v1:172.21.15.72:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f1674002020 con 0x7f167c0ae930 2024-03-29T11:45:27.741 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.732+0000 7f167b59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:27.741 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.732+0000 7f167b59e640 4 set_mon_vals no callback set 2024-03-29T11:45:27.741 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.732+0000 7f167959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:27.741 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.732+0000 7f167959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:45:27.741 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.732+0000 7f167959a640 10 monclient: dump: 2024-03-29T11:45:27.741 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:45:27.741 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:45:27.741 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:27.741 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:27.741 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:27.741 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:27.742 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:45:27.742 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:45:27.742 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:45:27.742 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:27.742 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.733+0000 7f167b59e640 10 monclient: shutdown 2024-03-29T11:45:27.742 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.733+0000 7f167b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:27.742 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.733+0000 7f167b59e640 1 -- 172.21.15.72:0/1463077548 >> v1:172.21.15.72:6789/0 conn(0x7f167c0ae930 legacy=0x7f167c0ac3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:27.742 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.733+0000 7f167b59e640 1 -- 172.21.15.72:0/1463077548 shutdown_connections 2024-03-29T11:45:27.742 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.733+0000 7f167b59e640 1 -- 172.21.15.72:0/1463077548 >> 172.21.15.72:0/1463077548 conn(0x7f167c0673e0 msgr2=0x7f167c0b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:27.742 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.737+0000 7f167b59e640 1 -- 172.21.15.72:0/1463077548 shutdown_connections 2024-03-29T11:45:27.743 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.737+0000 7f167b59e640 1 -- 172.21.15.72:0/1463077548 wait complete. 2024-03-29T11:45:27.744 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.739+0000 7f167b59e640 10 monclient: build_initial_monmap 2024-03-29T11:45:27.744 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.739+0000 7f167b59e640 10 monclient: monmap: 2024-03-29T11:45:27.744 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:45:27.744 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:27.744 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:45:27.744 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:45:27.744 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:27.744 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:27.744 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:45:27.745 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:45:27.745 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:45:27.745 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:27.745 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.739+0000 7f167b59e640 1 librados: starting msgr at 2024-03-29T11:45:27.745 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.739+0000 7f167b59e640 1 librados: starting objecter 2024-03-29T11:45:27.745 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.739+0000 7f167b59e640 1 Processor -- start 2024-03-29T11:45:27.745 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.740+0000 7f167b59e640 1 -- start start 2024-03-29T11:45:27.745 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.740+0000 7f167b59e640 1 librados: setting wanted keys 2024-03-29T11:45:27.745 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.740+0000 7f167b59e640 1 librados: calling monclient init 2024-03-29T11:45:27.745 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.740+0000 7f167b59e640 10 monclient: init 2024-03-29T11:45:27.746 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.741+0000 7f167b59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:27.746 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.741+0000 7f167b59e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:45:27.746 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.741+0000 7f167b59e640 10 monclient(hunting): picked mon.noname-c con 0x7f167c0a9750 addr v1:172.21.15.72:6790/0 2024-03-29T11:45:27.746 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.741+0000 7f167b59e640 10 monclient(hunting): picked mon.noname-a con 0x7f167c0acab0 addr v1:172.21.15.72:6789/0 2024-03-29T11:45:27.746 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.741+0000 7f167b59e640 10 monclient(hunting): picked mon.noname-b con 0x7f167c0ae930 addr v1:172.21.15.89:6789/0 2024-03-29T11:45:27.754 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.744+0000 7f167b59e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f167c1526b0 con 0x7f167c0acab0 2024-03-29T11:45:27.754 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.744+0000 7f167b59e640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f167c07bd80 con 0x7f167c0ae930 2024-03-29T11:45:27.754 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.744+0000 7f167b59e640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f167c14c310 con 0x7f167c0a9750 2024-03-29T11:45:27.754 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.744+0000 7f167b59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:27.754 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.744+0000 7f167b59e640 10 monclient(hunting): authenticate will time out at 1487.032784s 2024-03-29T11:45:27.754 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.744+0000 7f167ad9d640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f167c0ae930 0x7f167c14bc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:52769/0 (socket says 172.21.15.72:52769) 2024-03-29T11:45:27.754 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.744+0000 7f167ad9d640 1 -- 172.21.15.72:0/1370867124 learned_addr learned my addr 172.21.15.72:0/1370867124 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:45:27.754 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.745+0000 7f166affd640 10 client.?.objecter ms_handle_connect 0x7f167c0ae930 2024-03-29T11:45:27.754 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.745+0000 7f166affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:27.754 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.745+0000 7f166affd640 10 client.?.objecter ms_handle_connect 0x7f167c0a9750 2024-03-29T11:45:27.755 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.745+0000 7f166affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:27.755 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.745+0000 7f166affd640 1 -- 172.21.15.72:0/1370867124 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3411567954 0 0) 0x7f167c07bd80 con 0x7f167c0ae930 2024-03-29T11:45:27.755 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.745+0000 7f166affd640 10 monclient(hunting): _negotiate 2024-03-29T11:45:27.755 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.745+0000 7f166affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:27.755 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.745+0000 7f166affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:27.755 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.745+0000 7f166affd640 10 monclient(hunting): my global_id is 4543 2024-03-29T11:45:27.755 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.745+0000 7f166affd640 1 -- 172.21.15.72:0/1370867124 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f165c004eb0 con 0x7f167c0ae930 2024-03-29T11:45:27.755 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.746+0000 7f166affd640 10 client.?.objecter ms_handle_connect 0x7f167c0acab0 2024-03-29T11:45:27.755 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.746+0000 7f166affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:27.755 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.746+0000 7f166affd640 1 -- 172.21.15.72:0/1370867124 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1361217832 0 0) 0x7f167c1526b0 con 0x7f167c0acab0 2024-03-29T11:45:27.755 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.746+0000 7f166affd640 10 monclient(hunting): _negotiate 2024-03-29T11:45:27.755 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.746+0000 7f166affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:27.755 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.746+0000 7f166affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:27.755 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.746+0000 7f166affd640 10 monclient(hunting): my global_id is 4593 2024-03-29T11:45:27.756 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.746+0000 7f166affd640 1 -- 172.21.15.72:0/1370867124 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f167c07bd80 con 0x7f167c0acab0 2024-03-29T11:45:27.756 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.746+0000 7f166affd640 1 -- 172.21.15.72:0/1370867124 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 334734853 0 0) 0x7f165c004eb0 con 0x7f167c0ae930 2024-03-29T11:45:27.756 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.746+0000 7f166affd640 1 -- 172.21.15.72:0/1370867124 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f167c1526b0 con 0x7f167c0ae930 2024-03-29T11:45:27.756 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.746+0000 7f166affd640 1 -- 172.21.15.72:0/1370867124 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f16700018c0 con 0x7f167c0ae930 2024-03-29T11:45:27.756 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.746+0000 7f166affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:27.756 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.746+0000 7f166affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:45:27.756 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.747+0000 7f166affd640 10 monclient(hunting): dump: 2024-03-29T11:45:27.756 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:45:27.756 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:45:27.756 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:27.756 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:27.756 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:27.756 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:27.756 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:45:27.756 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:45:27.756 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:45:27.757 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:27.757 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.747+0000 7f166affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:27.757 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.747+0000 7f166affd640 1 -- 172.21.15.72:0/1370867124 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3261460195 0 0) 0x7f167c07bd80 con 0x7f167c0acab0 2024-03-29T11:45:27.757 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.747+0000 7f166affd640 1 -- 172.21.15.72:0/1370867124 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f165c004eb0 con 0x7f167c0acab0 2024-03-29T11:45:27.757 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.747+0000 7f166affd640 1 -- 172.21.15.72:0/1370867124 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f1674001fa0 con 0x7f167c0acab0 2024-03-29T11:45:27.757 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.747+0000 7f166affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:27.757 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.747+0000 7f166affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:45:27.757 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.747+0000 7f166affd640 10 monclient(hunting): dump: 2024-03-29T11:45:27.757 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:45:27.757 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:45:27.757 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:27.757 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:27.757 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:27.757 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:27.757 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:45:27.757 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:45:27.758 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:45:27.758 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:27.758 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.747+0000 7f166affd640 1 -- 172.21.15.72:0/1370867124 <== mon.0 v1:172.21.15.72:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2913351122 0 0) 0x7f165c004eb0 con 0x7f167c0acab0 2024-03-29T11:45:27.758 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.747+0000 7f166affd640 1 -- 172.21.15.72:0/1370867124 >> v1:172.21.15.72:6790/0 conn(0x7f167c0a9750 legacy=0x7f167c14ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:27.758 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.747+0000 7f166affd640 1 -- 172.21.15.72:0/1370867124 >> v1:172.21.15.89:6789/0 conn(0x7f167c0ae930 legacy=0x7f167c14bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:27.758 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.747+0000 7f166affd640 10 monclient: _finish_hunting 0 2024-03-29T11:45:27.758 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.747+0000 7f166affd640 1 monclient: found mon.a 2024-03-29T11:45:27.758 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.747+0000 7f166affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:27.758 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.748+0000 7f166affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:45:27.758 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.748+0000 7f166affd640 1 -- 172.21.15.72:0/1370867124 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f167c14c590 con 0x7f167c0acab0 2024-03-29T11:45:27.758 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.748+0000 7f166affd640 10 monclient: _finish_auth 0 2024-03-29T11:45:27.759 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.748+0000 7f166affd640 10 monclient: _check_auth_tickets 2024-03-29T11:45:27.759 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.748+0000 7f166affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:27.759 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.748+0000 7f167b59e640 5 monclient: authenticate success, global_id 4593 2024-03-29T11:45:27.759 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.748+0000 7f167b59e640 10 monclient: _renew_subs 2024-03-29T11:45:27.759 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.748+0000 7f167b59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:45:27.759 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.748+0000 7f167b59e640 1 -- 172.21.15.72:0/1370867124 --> v1:172.21.15.72:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f167c1506f0 con 0x7f167c0acab0 2024-03-29T11:45:27.759 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.748+0000 7f167b59e640 10 client.4593.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:27.759 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.748+0000 7f167b59e640 10 monclient: _renew_subs 2024-03-29T11:45:27.759 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.748+0000 7f167b59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:45:27.759 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.748+0000 7f167b59e640 1 -- 172.21.15.72:0/1370867124 --> v1:172.21.15.72:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f167c14eb30 con 0x7f167c0acab0 2024-03-29T11:45:27.759 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.748+0000 7f167b59e640 1 librados: init done 2024-03-29T11:45:27.761 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.750+0000 7f1668ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:27.761 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.750+0000 7f1668ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:27.761 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.750+0000 7f1668ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:45:27.761 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.750+0000 7f1668ff9640 1 -- 172.21.15.72:0/1370867124 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f167c1524a0 con 0x7f167c0acab0 2024-03-29T11:45:27.761 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.751+0000 7f166affd640 1 -- 172.21.15.72:0/1370867124 <== mon.0 v1:172.21.15.72:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1674004e10 con 0x7f167c0acab0 2024-03-29T11:45:27.761 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.751+0000 7f166affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:27.761 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.751+0000 7f1678d99640 4 set_mon_vals no callback set 2024-03-29T11:45:27.761 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.751+0000 7f166affd640 1 -- 172.21.15.72:0/1370867124 <== mon.0 v1:172.21.15.72:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f1674002100 con 0x7f167c0acab0 2024-03-29T11:45:27.761 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.751+0000 7f166affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:27.761 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.751+0000 7f166affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:45:27.761 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.751+0000 7f166affd640 10 monclient: dump: 2024-03-29T11:45:27.761 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:45:27.761 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:45:27.762 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:27.762 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:27.762 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:27.762 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:27.762 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:45:27.762 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:45:27.762 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:45:27.762 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:27.762 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.754+0000 7f166affd640 1 -- 172.21.15.72:0/1370867124 <== mon.0 v1:172.21.15.72:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f167401af90 con 0x7f167c0acab0 2024-03-29T11:45:27.762 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.754+0000 7f166affd640 1 -- 172.21.15.72:0/1370867124 <== mon.0 v1:172.21.15.72:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3662182158 0 0) 0x7f1674084db0 con 0x7f167c0acab0 2024-03-29T11:45:27.762 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.754+0000 7f167a59c640 10 monclient: get_auth_request con 0x7f165c06dee0 auth_method 0 2024-03-29T11:45:27.762 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.754+0000 7f166affd640 10 client.4593.objecter ms_dispatch 0x7f167c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:45:27.762 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.754+0000 7f166affd640 3 client.4593.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:45:27.762 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.754+0000 7f166affd640 3 client.4593.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:45:27.762 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.754+0000 7f166affd640 20 client.4593.objecter dump_active .. 0 homeless 2024-03-29T11:45:27.763 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.755+0000 7f166affd640 10 client.4593.objecter ms_handle_connect 0x7f165c06dee0 2024-03-29T11:45:27.767 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.761+0000 7f166affd640 1 -- 172.21.15.72:0/1370867124 <== mon.0 v1:172.21.15.72:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1674050690 con 0x7f167c0acab0 2024-03-29T11:45:27.767 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.761+0000 7f166affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:27.767 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:27.761+0000 7f166affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:28.091 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.087+0000 7f1668ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:28.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.087+0000 7f1668ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:28.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.087+0000 7f1668ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:45:28.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.087+0000 7f1668ff9640 1 -- 172.21.15.72:0/1370867124 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f167c0abd00 con 0x7f167c0acab0 2024-03-29T11:45:28.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.088+0000 7f1679d9b640 1 -- 172.21.15.72:0/1370867124 >> v1:172.21.15.72:6789/0 conn(0x7f167c0acab0 legacy=0x7f167c14b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-29T11:45:28.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.088+0000 7f1679d9b640 1 -- 172.21.15.72:0/1370867124 >> v1:172.21.15.72:6789/0 conn(0x7f167c0acab0 legacy=0x7f167c14b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:45:28.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.088+0000 7f1679d9b640 1 --1- 172.21.15.72:0/1370867124 >> v1:172.21.15.72:6789/0 conn(0x7f167c0acab0 0x7f167c14b4f0 :-1 s=OPENED pgs=1017116 cs=1 l=1).handle_message read tag failed 2024-03-29T11:45:28.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.088+0000 7f1679d9b640 1 --1- 172.21.15.72:0/1370867124 >> v1:172.21.15.72:6789/0 conn(0x7f167c0acab0 0x7f167c14b4f0 :-1 s=OPENED pgs=1017116 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:45:28.093 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.088+0000 7f166affd640 10 monclient: ms_handle_reset current mon v1:172.21.15.72:6789/0 2024-03-29T11:45:28.093 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.088+0000 7f166affd640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:28.093 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.088+0000 7f166affd640 1 -- 172.21.15.72:0/1370867124 >> v1:172.21.15.72:6789/0 conn(0x7f167c0acab0 legacy=0x7f167c14b4f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:45:28.093 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.088+0000 7f166affd640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:45:28.093 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.088+0000 7f166affd640 10 monclient(hunting): picked mon.a con 0x7f165c072ed0 addr v1:172.21.15.72:6789/0 2024-03-29T11:45:28.093 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.088+0000 7f166affd640 10 monclient(hunting): picked mon.c con 0x7f167c0a9750 addr v1:172.21.15.72:6790/0 2024-03-29T11:45:28.093 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.088+0000 7f166affd640 10 monclient(hunting): picked mon.b con 0x7f167c0ae930 addr v1:172.21.15.89:6789/0 2024-03-29T11:45:28.093 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.088+0000 7f166affd640 1 -- 172.21.15.72:0/1370867124 --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f165c004eb0 con 0x7f165c072ed0 2024-03-29T11:45:28.093 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.088+0000 7f166affd640 1 -- 172.21.15.72:0/1370867124 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f167c07bd80 con 0x7f167c0ae930 2024-03-29T11:45:28.093 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.088+0000 7f166affd640 1 -- 172.21.15.72:0/1370867124 --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f165c0806b0 con 0x7f167c0a9750 2024-03-29T11:45:28.093 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.088+0000 7f166affd640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:28.095 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.091+0000 7f166affd640 10 client.4593.objecter ms_handle_connect 0x7f165c072ed0 2024-03-29T11:45:28.096 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.091+0000 7f166affd640 10 client.4593.objecter resend_mon_ops 2024-03-29T11:45:28.096 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.092+0000 7f166affd640 10 client.4593.objecter ms_handle_connect 0x7f167c0a9750 2024-03-29T11:45:28.096 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.092+0000 7f166affd640 10 client.4593.objecter resend_mon_ops 2024-03-29T11:45:28.097 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.093+0000 7f166affd640 1 -- 172.21.15.72:0/1370867124 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1641755779 0 0) 0x7f165c004eb0 con 0x7f165c072ed0 2024-03-29T11:45:28.097 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.093+0000 7f166affd640 10 monclient(hunting): _negotiate 2024-03-29T11:45:28.097 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.093+0000 7f166affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:28.097 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.093+0000 7f166affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T11:45:28.097 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.093+0000 7f166affd640 1 -- 172.21.15.72:0/1370867124 --> v1:172.21.15.72:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f165c080840 con 0x7f165c072ed0 2024-03-29T11:45:28.099 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.095+0000 7f166affd640 1 -- 172.21.15.72:0/1370867124 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 1519104394 0 0) 0x7f165c080840 con 0x7f165c072ed0 2024-03-29T11:45:28.099 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.095+0000 7f166affd640 1 -- 172.21.15.72:0/1370867124 >> v1:172.21.15.72:6790/0 conn(0x7f167c0a9750 legacy=0x7f165c078de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:28.099 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.095+0000 7f166affd640 1 -- 172.21.15.72:0/1370867124 >> v1:172.21.15.89:6789/0 conn(0x7f167c0ae930 legacy=0x7f165c07ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:28.100 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.095+0000 7f166affd640 10 monclient: _finish_hunting 0 2024-03-29T11:45:28.100 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.095+0000 7f166affd640 1 monclient: found mon.a 2024-03-29T11:45:28.100 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.095+0000 7f166affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:28.100 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.095+0000 7f166affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:45:28.100 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.095+0000 7f166affd640 1 -- 172.21.15.72:0/1370867124 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f165c0818c0 con 0x7f165c072ed0 2024-03-29T11:45:28.100 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.095+0000 7f166affd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:28.100 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.095+0000 7f166affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:45:28.100 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.095+0000 7f166affd640 1 -- 172.21.15.72:0/1370867124 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f167c07bd80 con 0x7f165c072ed0 2024-03-29T11:45:28.100 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.095+0000 7f166affd640 10 monclient: _finish_auth 0 2024-03-29T11:45:28.100 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.095+0000 7f166affd640 10 monclient: _check_auth_tickets 2024-03-29T11:45:28.100 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.095+0000 7f166affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:28.100 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.095+0000 7f166affd640 10 client.4593.objecter ms_handle_connect 0x7f167c0ae930 2024-03-29T11:45:28.100 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.095+0000 7f166affd640 10 client.4593.objecter resend_mon_ops 2024-03-29T11:45:28.100 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.095+0000 7f166affd640 1 -- 172.21.15.72:0/1370867124 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f1674050690 con 0x7f165c072ed0 2024-03-29T11:45:28.100 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.095+0000 7f166affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:28.101 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.095+0000 7f166affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:45:28.101 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.095+0000 7f166affd640 10 monclient: dump: 2024-03-29T11:45:28.101 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:45:28.101 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:45:28.101 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:28.101 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:28.101 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:28.101 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:28.101 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:45:28.101 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:45:28.101 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:45:28.101 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:28.102 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.096+0000 7f166affd640 1 -- 172.21.15.72:0/1370867124 <== mon.0 v1:172.21.15.72:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16740525c0 con 0x7f165c072ed0 2024-03-29T11:45:28.102 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.096+0000 7f166affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:28.102 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.096+0000 7f1678d99640 4 set_mon_vals no callback set 2024-03-29T11:45:28.102 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.097+0000 7f166affd640 1 -- 172.21.15.72:0/1370867124 <== mon.0 v1:172.21.15.72:6789/0 5 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f1674050140 con 0x7f165c072ed0 2024-03-29T11:45:28.102 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.098+0000 7f166affd640 1 -- 172.21.15.72:0/1370867124 <== mon.0 v1:172.21.15.72:6789/0 6 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3214350691) 0x7f1674087510 con 0x7f165c072ed0 2024-03-29T11:45:28.103 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.098+0000 7f166affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:28.103 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.098+0000 7f166affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:28.103 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:45:28.103 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":11,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:45:05.043721+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:45:05.043721+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:28.109 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.102+0000 7f1668ff9640 10 librados: watch_flush enter 2024-03-29T11:45:28.109 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.104+0000 7f1668ff9640 10 librados: watch_flush exit 2024-03-29T11:45:28.109 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.104+0000 7f1668ff9640 20 client.4593.objecter shutdown clearing up homeless session... 2024-03-29T11:45:28.109 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.104+0000 7f1668ff9640 10 client.4593.objecter successfully canceled tick 2024-03-29T11:45:28.110 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.106+0000 7f1668ff9640 1 -- 172.21.15.72:0/1370867124 >> v1:172.21.15.72:6816/1578866102 conn(0x7f165c06dee0 legacy=0x7f165c0703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:28.110 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.106+0000 7f1668ff9640 10 monclient: shutdown 2024-03-29T11:45:28.110 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.106+0000 7f1668ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:28.110 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.106+0000 7f1668ff9640 1 -- 172.21.15.72:0/1370867124 >> v1:172.21.15.72:6789/0 conn(0x7f165c072ed0 legacy=0x7f165c0752e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:28.114 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.109+0000 7f1668ff9640 1 -- 172.21.15.72:0/1370867124 shutdown_connections 2024-03-29T11:45:28.114 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.109+0000 7f1668ff9640 1 -- 172.21.15.72:0/1370867124 >> 172.21.15.72:0/1370867124 conn(0x7f167c0673e0 msgr2=0x7f167c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:28.115 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.111+0000 7f1668ff9640 1 -- 172.21.15.72:0/1370867124 shutdown_connections 2024-03-29T11:45:28.116 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.111+0000 7f1668ff9640 1 -- 172.21.15.72:0/1370867124 wait complete. 2024-03-29T11:45:28.116 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:28.111+0000 7f1668ff9640 1 librados: shutdown 2024-03-29T11:45:28.134 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:34.940 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:35.082 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.077+0000 7f7fcda7d640 10 monclient: get_monmap_and_config 2024-03-29T11:45:35.082 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.077+0000 7f7fcda7d640 10 monclient: build_initial_monmap 2024-03-29T11:45:35.082 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.077+0000 7f7fcda7d640 10 monclient: monmap: 2024-03-29T11:45:35.082 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:45:35.082 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:45:35.082 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:45:35.078313+0000 2024-03-29T11:45:35.082 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:45:35.078313+0000 2024-03-29T11:45:35.082 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:35.082 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:35.082 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:45:35.082 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:45:35.083 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:45:35.083 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:35.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.078+0000 7f7fcda7d640 1 Processor -- start 2024-03-29T11:45:35.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.078+0000 7f7fcda7d640 1 -- start start 2024-03-29T11:45:35.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.078+0000 7f7fcda7d640 10 monclient: init 2024-03-29T11:45:35.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.079+0000 7f7fcda7d640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:35.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.079+0000 7f7fcda7d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:45:35.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.079+0000 7f7fcda7d640 10 monclient(hunting): picked mon.noname-b con 0x7f7fc81422b0 addr v1:172.21.15.72:6790/0 2024-03-29T11:45:35.085 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.079+0000 7f7fcda7d640 10 monclient(hunting): picked mon.noname-c con 0x7f7fc8142690 addr v1:172.21.15.89:6789/0 2024-03-29T11:45:35.086 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.079+0000 7f7fcda7d640 10 monclient(hunting): picked mon.noname-a con 0x7f7fc80b17d0 addr v1:172.21.15.72:6789/0 2024-03-29T11:45:35.086 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.079+0000 7f7fc77fe640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f7fc81422b0 0x7f7fc813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:37388/0 (socket says 172.21.15.72:37388) 2024-03-29T11:45:35.086 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.079+0000 7f7fc77fe640 1 -- 172.21.15.72:0/929310136 learned_addr learned my addr 172.21.15.72:0/929310136 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:45:35.086 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.080+0000 7f7fcda7d640 1 -- 172.21.15.72:0/929310136 --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc8142c80 con 0x7f7fc80b17d0 2024-03-29T11:45:35.086 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.080+0000 7f7fcda7d640 1 -- 172.21.15.72:0/929310136 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc80aae40 con 0x7f7fc8142690 2024-03-29T11:45:35.086 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.080+0000 7f7fcda7d640 1 -- 172.21.15.72:0/929310136 --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc813fbd0 con 0x7f7fc81422b0 2024-03-29T11:45:35.086 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.080+0000 7f7fcda7d640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:35.086 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.080+0000 7f7fcda7d640 10 monclient(hunting): authenticate will time out at 1494.368193s 2024-03-29T11:45:35.086 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.080+0000 7f7fc67fc640 1 -- 172.21.15.72:0/929310136 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1996972445 0 0) 0x7f7fc8142c80 con 0x7f7fc80b17d0 2024-03-29T11:45:35.086 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.080+0000 7f7fc67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:45:35.086 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.080+0000 7f7fc67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:35.086 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.080+0000 7f7fc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:35.086 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.080+0000 7f7fc67fc640 10 monclient(hunting): my global_id is 4596 2024-03-29T11:45:35.087 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.080+0000 7f7fc67fc640 1 -- 172.21.15.72:0/929310136 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fb0003520 con 0x7f7fc80b17d0 2024-03-29T11:45:35.087 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.080+0000 7f7fc67fc640 1 -- 172.21.15.72:0/929310136 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1057362766 0 0) 0x7f7fc813fbd0 con 0x7f7fc81422b0 2024-03-29T11:45:35.087 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.080+0000 7f7fc67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:45:35.087 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.080+0000 7f7fc67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:35.087 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.080+0000 7f7fc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:35.087 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.080+0000 7f7fc67fc640 10 monclient(hunting): my global_id is 4622 2024-03-29T11:45:35.087 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.080+0000 7f7fc67fc640 1 -- 172.21.15.72:0/929310136 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fc8142c80 con 0x7f7fc81422b0 2024-03-29T11:45:35.087 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.080+0000 7f7fc67fc640 1 -- 172.21.15.72:0/929310136 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1519012619 0 0) 0x7f7fb0003520 con 0x7f7fc80b17d0 2024-03-29T11:45:35.087 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.081+0000 7f7fc67fc640 1 -- 172.21.15.72:0/929310136 >> v1:172.21.15.72:6790/0 conn(0x7f7fc81422b0 legacy=0x7f7fc813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:35.087 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.081+0000 7f7fc67fc640 1 -- 172.21.15.72:0/929310136 >> v1:172.21.15.89:6789/0 conn(0x7f7fc8142690 legacy=0x7f7fc80b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:45:35.090 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.082+0000 7f7fc67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:45:35.090 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.082+0000 7f7fc67fc640 1 monclient: found mon.noname-a 2024-03-29T11:45:35.090 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.082+0000 7f7fc67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:35.090 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.082+0000 7f7fc67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.72:6789/0 2024-03-29T11:45:35.090 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.082+0000 7f7fc67fc640 1 -- 172.21.15.72:0/929310136 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc8151940 con 0x7f7fc80b17d0 2024-03-29T11:45:35.090 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.082+0000 7f7fc67fc640 10 monclient: _finish_auth 0 2024-03-29T11:45:35.090 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.082+0000 7f7fc67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:45:35.090 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.083+0000 7f7fc67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:35.090 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.083+0000 7f7fcda7d640 5 monclient: authenticate success, global_id 4596 2024-03-29T11:45:35.090 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.083+0000 7f7fcda7d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:35.090 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.083+0000 7f7fc67fc640 1 -- 172.21.15.72:0/929310136 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f7fc0001990 con 0x7f7fc80b17d0 2024-03-29T11:45:35.090 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.083+0000 7f7fc67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:35.090 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.083+0000 7f7fc67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:45:35.091 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.083+0000 7f7fc67fc640 10 monclient: dump: 2024-03-29T11:45:35.091 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:45:35.091 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:45:35.091 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:35.091 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:35.091 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:35.091 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:35.091 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:45:35.091 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:45:35.091 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:45:35.091 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:35.091 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.083+0000 7f7fcda7d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:35.091 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.083+0000 7f7fc67fc640 1 -- 172.21.15.72:0/929310136 <== mon.0 v1:172.21.15.72:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7fc0001bb0 con 0x7f7fc80b17d0 2024-03-29T11:45:35.091 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.083+0000 7f7fc67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:35.091 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.083+0000 7f7fcda7d640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:35.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.083+0000 7f7fcda7d640 4 set_mon_vals no callback set 2024-03-29T11:45:35.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.083+0000 7f7fc67fc640 1 -- 172.21.15.72:0/929310136 <== mon.0 v1:172.21.15.72:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f7fc0001f40 con 0x7f7fc80b17d0 2024-03-29T11:45:35.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.083+0000 7f7fc67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:35.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.083+0000 7f7fc67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:45:35.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.083+0000 7f7fc67fc640 10 monclient: dump: 2024-03-29T11:45:35.092 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:45:35.092 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:45:35.092 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:35.092 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:35.092 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:35.092 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:35.092 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:45:35.092 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:45:35.092 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:45:35.092 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:35.092 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.084+0000 7f7fcda7d640 10 monclient: shutdown 2024-03-29T11:45:35.093 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.084+0000 7f7fcda7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:35.093 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.084+0000 7f7fcda7d640 1 -- 172.21.15.72:0/929310136 >> v1:172.21.15.72:6789/0 conn(0x7f7fc80b17d0 legacy=0x7f7fc80b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:35.093 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.084+0000 7f7fcda7d640 1 -- 172.21.15.72:0/929310136 shutdown_connections 2024-03-29T11:45:35.093 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.084+0000 7f7fcda7d640 1 -- 172.21.15.72:0/929310136 >> 172.21.15.72:0/929310136 conn(0x7f7fc8066a00 msgr2=0x7f7fc8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:35.093 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.084+0000 7f7fcda7d640 1 -- 172.21.15.72:0/929310136 shutdown_connections 2024-03-29T11:45:35.093 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.084+0000 7f7fcda7d640 1 -- 172.21.15.72:0/929310136 wait complete. 2024-03-29T11:45:35.093 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.085+0000 7f7fcda7d640 10 monclient: build_initial_monmap 2024-03-29T11:45:35.093 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.085+0000 7f7fcda7d640 10 monclient: monmap: 2024-03-29T11:45:35.093 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:45:35.093 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:35.093 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:45:35.093 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:45:35.093 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:35.093 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:35.093 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:45:35.094 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:45:35.094 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:45:35.094 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:35.099 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.085+0000 7f7fcda7d640 1 librados: starting msgr at 2024-03-29T11:45:35.099 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.085+0000 7f7fcda7d640 1 librados: starting objecter 2024-03-29T11:45:35.099 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.086+0000 7f7fcda7d640 1 Processor -- start 2024-03-29T11:45:35.099 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.086+0000 7f7fcda7d640 1 -- start start 2024-03-29T11:45:35.099 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.086+0000 7f7fcda7d640 1 librados: setting wanted keys 2024-03-29T11:45:35.100 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.086+0000 7f7fcda7d640 1 librados: calling monclient init 2024-03-29T11:45:35.100 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.086+0000 7f7fcda7d640 10 monclient: init 2024-03-29T11:45:35.100 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.087+0000 7f7fcda7d640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:35.100 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.087+0000 7f7fcda7d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:45:35.100 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.087+0000 7f7fcda7d640 10 monclient(hunting): picked mon.noname-c con 0x7f7fc80b17d0 addr v1:172.21.15.72:6790/0 2024-03-29T11:45:35.100 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.087+0000 7f7fcda7d640 10 monclient(hunting): picked mon.noname-b con 0x7f7fc81422b0 addr v1:172.21.15.89:6789/0 2024-03-29T11:45:35.100 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.087+0000 7f7fcda7d640 10 monclient(hunting): picked mon.noname-a con 0x7f7fc8142690 addr v1:172.21.15.72:6789/0 2024-03-29T11:45:35.100 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.087+0000 7f7fcda7d640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc80b7090 con 0x7f7fc8142690 2024-03-29T11:45:35.100 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.087+0000 7f7fcda7d640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc806ef30 con 0x7f7fc81422b0 2024-03-29T11:45:35.100 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.087+0000 7f7fcda7d640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc80b7520 con 0x7f7fc80b17d0 2024-03-29T11:45:35.100 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.087+0000 7f7fcda7d640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:35.100 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.087+0000 7f7fcda7d640 10 monclient(hunting): authenticate will time out at 1494.375213s 2024-03-29T11:45:35.100 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.087+0000 7f7fc77fe640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f7fc80b17d0 0x7f7fc81dec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:60758/0 (socket says 172.21.15.72:60758) 2024-03-29T11:45:35.100 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.087+0000 7f7fc77fe640 1 -- 172.21.15.72:0/3048850410 learned_addr learned my addr 172.21.15.72:0/3048850410 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:45:35.101 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.088+0000 7f7faffff640 10 client.?.objecter ms_handle_connect 0x7f7fc80b17d0 2024-03-29T11:45:35.101 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.088+0000 7f7faffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:35.101 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.088+0000 7f7faffff640 10 client.?.objecter ms_handle_connect 0x7f7fc8142690 2024-03-29T11:45:35.101 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.088+0000 7f7faffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:35.101 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.088+0000 7f7faffff640 1 -- 172.21.15.72:0/3048850410 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 774649094 0 0) 0x7f7fc80b7520 con 0x7f7fc80b17d0 2024-03-29T11:45:35.101 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.088+0000 7f7faffff640 10 monclient(hunting): _negotiate 2024-03-29T11:45:35.101 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.088+0000 7f7faffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:35.101 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.088+0000 7f7faffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:35.101 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.088+0000 7f7faffff640 10 monclient(hunting): my global_id is 4625 2024-03-29T11:45:35.101 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.088+0000 7f7faffff640 1 -- 172.21.15.72:0/3048850410 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fb4004eb0 con 0x7f7fc80b17d0 2024-03-29T11:45:35.101 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.088+0000 7f7faffff640 1 -- 172.21.15.72:0/3048850410 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1623954003 0 0) 0x7f7fc80b7090 con 0x7f7fc8142690 2024-03-29T11:45:35.101 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.088+0000 7f7faffff640 10 monclient(hunting): _negotiate 2024-03-29T11:45:35.101 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.088+0000 7f7faffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:35.101 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.088+0000 7f7faffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:35.101 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.088+0000 7f7faffff640 10 monclient(hunting): my global_id is 4599 2024-03-29T11:45:35.102 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.088+0000 7f7faffff640 1 -- 172.21.15.72:0/3048850410 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fc80b7520 con 0x7f7fc8142690 2024-03-29T11:45:35.102 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.088+0000 7f7faffff640 1 -- 172.21.15.72:0/3048850410 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1712824208 0 0) 0x7f7fc80b7520 con 0x7f7fc8142690 2024-03-29T11:45:35.102 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.088+0000 7f7faffff640 1 -- 172.21.15.72:0/3048850410 --> v1:172.21.15.72:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fc80b7090 con 0x7f7fc8142690 2024-03-29T11:45:35.102 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.088+0000 7f7faffff640 1 -- 172.21.15.72:0/3048850410 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f7fc00018d0 con 0x7f7fc8142690 2024-03-29T11:45:35.102 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.088+0000 7f7faffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:35.102 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.088+0000 7f7faffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:45:35.102 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.088+0000 7f7faffff640 10 monclient(hunting): dump: 2024-03-29T11:45:35.102 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:45:35.102 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:45:35.102 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:35.102 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:35.102 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:35.102 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:35.102 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:45:35.103 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:45:35.103 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:45:35.103 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:35.103 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.088+0000 7f7faffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:35.103 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.088+0000 7f7faffff640 1 -- 172.21.15.72:0/3048850410 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 77013776 0 0) 0x7f7fb4004eb0 con 0x7f7fc80b17d0 2024-03-29T11:45:35.103 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.089+0000 7f7faffff640 1 -- 172.21.15.72:0/3048850410 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fc80b7520 con 0x7f7fc80b17d0 2024-03-29T11:45:35.103 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.089+0000 7f7faffff640 1 -- 172.21.15.72:0/3048850410 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f7fbc001930 con 0x7f7fc80b17d0 2024-03-29T11:45:35.103 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.089+0000 7f7faffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:35.103 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.089+0000 7f7faffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:35.103 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.089+0000 7f7faffff640 10 monclient(hunting): dump: 2024-03-29T11:45:35.103 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:45:35.103 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:45:35.103 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:35.103 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:35.103 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:35.103 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:35.104 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:45:35.104 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:45:35.104 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:45:35.104 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:35.104 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.089+0000 7f7faffff640 1 -- 172.21.15.72:0/3048850410 <== mon.0 v1:172.21.15.72:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 954531386 0 0) 0x7f7fc80b7090 con 0x7f7fc8142690 2024-03-29T11:45:35.104 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.089+0000 7f7faffff640 1 -- 172.21.15.72:0/3048850410 >> v1:172.21.15.72:6790/0 conn(0x7f7fc80b17d0 legacy=0x7f7fc81dec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:35.104 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.089+0000 7f7faffff640 1 -- 172.21.15.72:0/3048850410 >> v1:172.21.15.89:6789/0 conn(0x7f7fc81422b0 legacy=0x7f7fc812ec20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:45:35.104 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.090+0000 7f7faffff640 10 monclient: _finish_hunting 0 2024-03-29T11:45:35.104 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.090+0000 7f7faffff640 1 monclient: found mon.a 2024-03-29T11:45:35.104 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.090+0000 7f7faffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:35.104 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.090+0000 7f7faffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:45:35.104 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.090+0000 7f7faffff640 1 -- 172.21.15.72:0/3048850410 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc8071c60 con 0x7f7fc8142690 2024-03-29T11:45:35.104 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.090+0000 7f7faffff640 10 monclient: _finish_auth 0 2024-03-29T11:45:35.104 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.090+0000 7f7faffff640 10 monclient: _check_auth_tickets 2024-03-29T11:45:35.104 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.090+0000 7f7faffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:35.105 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.090+0000 7f7fcda7d640 5 monclient: authenticate success, global_id 4599 2024-03-29T11:45:35.105 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.090+0000 7f7fcda7d640 10 monclient: _renew_subs 2024-03-29T11:45:35.105 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.090+0000 7f7fcda7d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:45:35.105 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.090+0000 7f7fcda7d640 1 -- 172.21.15.72:0/3048850410 --> v1:172.21.15.72:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fc8071ed0 con 0x7f7fc8142690 2024-03-29T11:45:35.105 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.090+0000 7f7fcda7d640 10 client.4599.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:35.105 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.090+0000 7f7fcda7d640 10 monclient: _renew_subs 2024-03-29T11:45:35.105 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.090+0000 7f7fcda7d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:45:35.105 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.090+0000 7f7fcda7d640 1 -- 172.21.15.72:0/3048850410 --> v1:172.21.15.72:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fc81df540 con 0x7f7fc8142690 2024-03-29T11:45:35.105 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.090+0000 7f7fcda7d640 1 librados: init done 2024-03-29T11:45:35.105 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.090+0000 7f7faffff640 1 -- 172.21.15.72:0/3048850410 <== mon.0 v1:172.21.15.72:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7fc0001a30 con 0x7f7fc8142690 2024-03-29T11:45:35.105 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.091+0000 7f7faffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:35.105 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.091+0000 7f7fc5ffb640 4 set_mon_vals no callback set 2024-03-29T11:45:35.105 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.092+0000 7f7faffff640 1 -- 172.21.15.72:0/3048850410 <== mon.0 v1:172.21.15.72:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f7fc0003c20 con 0x7f7fc8142690 2024-03-29T11:45:35.105 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.092+0000 7f7faffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:35.105 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.092+0000 7f7faffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:45:35.106 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.092+0000 7f7faffff640 10 monclient: dump: 2024-03-29T11:45:35.106 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:45:35.106 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:45:35.106 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:35.106 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:35.106 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:35.106 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:35.106 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:45:35.106 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:45:35.106 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:45:35.106 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:35.106 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.093+0000 7f7faffff640 1 -- 172.21.15.72:0/3048850410 <== mon.0 v1:172.21.15.72:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f7fc0003d80 con 0x7f7fc8142690 2024-03-29T11:45:35.106 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.093+0000 7f7fc77fe640 10 monclient: get_auth_request con 0x7f7fb406dee0 auth_method 0 2024-03-29T11:45:35.106 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.093+0000 7f7faffff640 1 -- 172.21.15.72:0/3048850410 <== mon.0 v1:172.21.15.72:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3662182158 0 0) 0x7f7fc0083b70 con 0x7f7fc8142690 2024-03-29T11:45:35.106 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.094+0000 7f7faffff640 10 client.4599.objecter ms_dispatch 0x7f7fc8000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:45:35.107 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.094+0000 7f7faffff640 3 client.4599.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:45:35.107 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.094+0000 7f7faffff640 3 client.4599.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:45:35.107 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.094+0000 7f7faffff640 20 client.4599.objecter dump_active .. 0 homeless 2024-03-29T11:45:35.107 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.094+0000 7f7faffff640 10 client.4599.objecter ms_handle_connect 0x7f7fb406dee0 2024-03-29T11:45:35.107 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.094+0000 7f7fcda7d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:35.107 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.094+0000 7f7fcda7d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:35.107 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.094+0000 7f7fcda7d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:45:35.107 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.094+0000 7f7fcda7d640 1 -- 172.21.15.72:0/3048850410 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fa8005080 con 0x7f7fc8142690 2024-03-29T11:45:35.108 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.104+0000 7f7faffff640 1 -- 172.21.15.72:0/3048850410 <== mon.0 v1:172.21.15.72:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7fc004f450 con 0x7f7fc8142690 2024-03-29T11:45:35.108 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.104+0000 7f7faffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:35.108 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.104+0000 7f7faffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:35.432 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.427+0000 7f7fcda7d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:35.432 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.427+0000 7f7fcda7d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:35.432 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.427+0000 7f7fcda7d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.72:6789/0 2024-03-29T11:45:35.432 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.427+0000 7f7fcda7d640 1 -- 172.21.15.72:0/3048850410 --> v1:172.21.15.72:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7fa8005330 con 0x7f7fc8142690 2024-03-29T11:45:35.434 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.429+0000 7f7faffff640 1 -- 172.21.15.72:0/3048850410 <== mon.0 v1:172.21.15.72:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3214350691) 0x7f7fc0051380 con 0x7f7fc8142690 2024-03-29T11:45:35.434 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.429+0000 7f7faffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:35.434 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.429+0000 7f7faffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:35.434 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:45:35.435 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":11,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:45:05.043721+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:45:05.043721+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:35.438 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.433+0000 7f7fcda7d640 10 librados: watch_flush enter 2024-03-29T11:45:35.438 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.433+0000 7f7fcda7d640 10 librados: watch_flush exit 2024-03-29T11:45:35.438 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.433+0000 7f7fcda7d640 20 client.4599.objecter shutdown clearing up homeless session... 2024-03-29T11:45:35.438 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.433+0000 7f7fcda7d640 10 client.4599.objecter successfully canceled tick 2024-03-29T11:45:35.438 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.433+0000 7f7fcda7d640 1 -- 172.21.15.72:0/3048850410 >> v1:172.21.15.72:6816/1578866102 conn(0x7f7fb406dee0 legacy=0x7f7fb40703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:35.438 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.433+0000 7f7fcda7d640 10 monclient: shutdown 2024-03-29T11:45:35.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.433+0000 7f7fcda7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:35.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.433+0000 7f7fcda7d640 1 -- 172.21.15.72:0/3048850410 >> v1:172.21.15.72:6789/0 conn(0x7f7fc8142690 legacy=0x7f7fc81e6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:35.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.433+0000 7f7fcda7d640 1 -- 172.21.15.72:0/3048850410 shutdown_connections 2024-03-29T11:45:35.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.433+0000 7f7fcda7d640 1 -- 172.21.15.72:0/3048850410 >> 172.21.15.72:0/3048850410 conn(0x7f7fc8066a00 msgr2=0x7f7fc813c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:35.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.434+0000 7f7fcda7d640 1 -- 172.21.15.72:0/3048850410 shutdown_connections 2024-03-29T11:45:35.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.434+0000 7f7fcda7d640 1 -- 172.21.15.72:0/3048850410 wait complete. 2024-03-29T11:45:35.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:35.435+0000 7f7fcda7d640 1 librados: shutdown 2024-03-29T11:45:35.462 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:42.269 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:42.423 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.418+0000 7f804cdc6640 10 monclient: get_monmap_and_config 2024-03-29T11:45:42.423 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.418+0000 7f804cdc6640 10 monclient: build_initial_monmap 2024-03-29T11:45:42.423 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.418+0000 7f804cdc6640 10 monclient: monmap: 2024-03-29T11:45:42.423 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:45:42.423 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:45:42.423 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:45:42.419499+0000 2024-03-29T11:45:42.423 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:45:42.419499+0000 2024-03-29T11:45:42.423 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:42.423 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:42.423 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:45:42.423 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:45:42.423 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:45:42.424 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:42.426 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.419+0000 7f804cdc6640 1 Processor -- start 2024-03-29T11:45:42.426 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.419+0000 7f804cdc6640 1 -- start start 2024-03-29T11:45:42.426 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.419+0000 7f804cdc6640 10 monclient: init 2024-03-29T11:45:42.427 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.420+0000 7f804cdc6640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:42.427 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.420+0000 7f804cdc6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:45:42.427 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.420+0000 7f804cdc6640 10 monclient(hunting): picked mon.noname-b con 0x7f804806cac0 addr v1:172.21.15.72:6790/0 2024-03-29T11:45:42.427 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.420+0000 7f804cdc6640 10 monclient(hunting): picked mon.noname-a con 0x7f804806cea0 addr v1:172.21.15.72:6789/0 2024-03-29T11:45:42.427 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.420+0000 7f804cdc6640 10 monclient(hunting): picked mon.noname-c con 0x7f8048145e80 addr v1:172.21.15.89:6789/0 2024-03-29T11:45:42.427 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.420+0000 7f804cdc6640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80480ac850 con 0x7f804806cea0 2024-03-29T11:45:42.427 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.420+0000 7f804cdc6640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8048070470 con 0x7f8048145e80 2024-03-29T11:45:42.427 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.420+0000 7f804cdc6640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8048072600 con 0x7f804806cac0 2024-03-29T11:45:42.427 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.420+0000 7f804cdc6640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:42.427 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.420+0000 7f804cdc6640 10 monclient(hunting): authenticate will time out at 1501.708589s 2024-03-29T11:45:42.427 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.420+0000 7f8045d9b640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f804806cea0 0x7f8048145770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:54576/0 (socket says 172.21.15.72:54576) 2024-03-29T11:45:42.427 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.420+0000 7f8045d9b640 1 -- 172.21.15.72:0/1525729458 learned_addr learned my addr 172.21.15.72:0/1525729458 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:45:42.427 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.421+0000 7f804559a640 1 -- 172.21.15.72:0/1525729458 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1820061019 0 0) 0x7f80480ac850 con 0x7f804806cea0 2024-03-29T11:45:42.427 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.421+0000 7f804559a640 10 monclient(hunting): _negotiate 2024-03-29T11:45:42.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.421+0000 7f804559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:42.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.421+0000 7f804559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:42.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.421+0000 7f804559a640 10 monclient(hunting): my global_id is 4602 2024-03-29T11:45:42.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.421+0000 7f804559a640 1 -- 172.21.15.72:0/1525729458 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8030003520 con 0x7f804806cea0 2024-03-29T11:45:42.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.421+0000 7f804559a640 1 -- 172.21.15.72:0/1525729458 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 62952588 0 0) 0x7f8048070470 con 0x7f8048145e80 2024-03-29T11:45:42.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.421+0000 7f804559a640 10 monclient(hunting): _negotiate 2024-03-29T11:45:42.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.421+0000 7f804559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:42.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.421+0000 7f804559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:42.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.421+0000 7f804559a640 10 monclient(hunting): my global_id is 4546 2024-03-29T11:45:42.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.421+0000 7f804559a640 1 -- 172.21.15.72:0/1525729458 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80480ac850 con 0x7f8048145e80 2024-03-29T11:45:42.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.422+0000 7f804559a640 1 -- 172.21.15.72:0/1525729458 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1931125013 0 0) 0x7f8030003520 con 0x7f804806cea0 2024-03-29T11:45:42.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.422+0000 7f804559a640 1 -- 172.21.15.72:0/1525729458 >> v1:172.21.15.72:6790/0 conn(0x7f804806cac0 legacy=0x7f8048071ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:42.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.422+0000 7f804559a640 1 -- 172.21.15.72:0/1525729458 >> v1:172.21.15.89:6789/0 conn(0x7f8048145e80 legacy=0x7f804814db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:42.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.422+0000 7f804559a640 10 monclient: _finish_hunting 0 2024-03-29T11:45:42.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.422+0000 7f804559a640 1 monclient: found mon.noname-a 2024-03-29T11:45:42.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.422+0000 7f804559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:42.428 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.422+0000 7f804559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.72:6789/0 2024-03-29T11:45:42.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.422+0000 7f804559a640 1 -- 172.21.15.72:0/1525729458 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80481462d0 con 0x7f804806cea0 2024-03-29T11:45:42.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.422+0000 7f804559a640 10 monclient: _finish_auth 0 2024-03-29T11:45:42.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.422+0000 7f804559a640 10 monclient: _check_auth_tickets 2024-03-29T11:45:42.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.422+0000 7f804559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:42.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.422+0000 7f804cdc6640 5 monclient: authenticate success, global_id 4602 2024-03-29T11:45:42.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.422+0000 7f804cdc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:42.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.422+0000 7f804559a640 1 -- 172.21.15.72:0/1525729458 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f803c001690 con 0x7f804806cea0 2024-03-29T11:45:42.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.422+0000 7f804559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:42.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.422+0000 7f804559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:45:42.429 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.422+0000 7f804559a640 10 monclient: dump: 2024-03-29T11:45:42.429 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:45:42.429 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:45:42.429 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:42.429 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:42.429 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:42.429 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:42.430 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:45:42.430 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:45:42.430 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:45:42.430 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:42.430 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.422+0000 7f804cdc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:42.438 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.423+0000 7f804559a640 1 -- 172.21.15.72:0/1525729458 <== mon.0 v1:172.21.15.72:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f803c001870 con 0x7f804806cea0 2024-03-29T11:45:42.438 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.423+0000 7f804559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:42.438 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.423+0000 7f804cdc6640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:42.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.423+0000 7f804cdc6640 4 set_mon_vals no callback set 2024-03-29T11:45:42.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.423+0000 7f804559a640 1 -- 172.21.15.72:0/1525729458 <== mon.0 v1:172.21.15.72:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f803c001690 con 0x7f804806cea0 2024-03-29T11:45:42.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.423+0000 7f804559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:42.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.423+0000 7f804559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:45:42.439 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.423+0000 7f804559a640 10 monclient: dump: 2024-03-29T11:45:42.439 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:45:42.439 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:45:42.439 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:42.439 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:42.439 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:42.439 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:42.439 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:45:42.439 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:45:42.440 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:45:42.440 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:42.440 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.424+0000 7f804cdc6640 10 monclient: shutdown 2024-03-29T11:45:42.440 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.424+0000 7f804cdc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:42.440 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.424+0000 7f804cdc6640 1 -- 172.21.15.72:0/1525729458 >> v1:172.21.15.72:6789/0 conn(0x7f804806cea0 legacy=0x7f8048145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:42.440 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.424+0000 7f804cdc6640 1 -- 172.21.15.72:0/1525729458 shutdown_connections 2024-03-29T11:45:42.440 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.424+0000 7f804cdc6640 1 -- 172.21.15.72:0/1525729458 >> 172.21.15.72:0/1525729458 conn(0x7f80480675b0 msgr2=0x7f80480b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:42.440 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.424+0000 7f804cdc6640 1 -- 172.21.15.72:0/1525729458 shutdown_connections 2024-03-29T11:45:42.440 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.424+0000 7f804cdc6640 1 -- 172.21.15.72:0/1525729458 wait complete. 2024-03-29T11:45:42.440 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.425+0000 7f804cdc6640 10 monclient: build_initial_monmap 2024-03-29T11:45:42.440 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.425+0000 7f804cdc6640 10 monclient: monmap: 2024-03-29T11:45:42.440 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:45:42.440 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:42.440 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:45:42.440 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:45:42.440 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:42.441 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:42.441 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:45:42.441 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:45:42.441 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:45:42.441 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:42.441 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.425+0000 7f804cdc6640 1 librados: starting msgr at 2024-03-29T11:45:42.441 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.425+0000 7f804cdc6640 1 librados: starting objecter 2024-03-29T11:45:42.441 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.426+0000 7f804cdc6640 1 Processor -- start 2024-03-29T11:45:42.441 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.426+0000 7f804cdc6640 1 -- start start 2024-03-29T11:45:42.441 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.426+0000 7f804cdc6640 1 librados: setting wanted keys 2024-03-29T11:45:42.441 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.426+0000 7f804cdc6640 1 librados: calling monclient init 2024-03-29T11:45:42.441 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.426+0000 7f804cdc6640 10 monclient: init 2024-03-29T11:45:42.441 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.426+0000 7f804cdc6640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:42.442 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.426+0000 7f804cdc6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:45:42.442 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.427+0000 7f804cdc6640 10 monclient(hunting): picked mon.noname-a con 0x7f804806cac0 addr v1:172.21.15.72:6789/0 2024-03-29T11:45:42.442 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.427+0000 7f804cdc6640 10 monclient(hunting): picked mon.noname-b con 0x7f804806cea0 addr v1:172.21.15.89:6789/0 2024-03-29T11:45:42.442 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.427+0000 7f804cdc6640 10 monclient(hunting): picked mon.noname-c con 0x7f8048145e80 addr v1:172.21.15.72:6790/0 2024-03-29T11:45:42.442 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.427+0000 7f804cdc6640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80480abf40 con 0x7f804806cac0 2024-03-29T11:45:42.442 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.427+0000 7f804cdc6640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80480ac220 con 0x7f804806cea0 2024-03-29T11:45:42.442 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.427+0000 7f804cdc6640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80480acf50 con 0x7f8048145e80 2024-03-29T11:45:42.442 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.427+0000 7f804cdc6640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:42.442 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.427+0000 7f804cdc6640 10 monclient(hunting): authenticate will time out at 1501.715207s 2024-03-29T11:45:42.442 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.427+0000 7f8046d9d640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f8048145e80 0x7f80481e2170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:51784/0 (socket says 172.21.15.72:51784) 2024-03-29T11:45:42.442 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.427+0000 7f8045d9b640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f804806cea0 0x7f80480aaad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:45820/0 (socket says 172.21.15.72:45820) 2024-03-29T11:45:42.443 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.427+0000 7f8045d9b640 1 -- 172.21.15.72:0/2444889081 learned_addr learned my addr 172.21.15.72:0/2444889081 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:45:42.443 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.427+0000 7f8036ffd640 10 client.?.objecter ms_handle_connect 0x7f804806cea0 2024-03-29T11:45:42.443 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.427+0000 7f8036ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:42.443 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.427+0000 7f8036ffd640 10 client.?.objecter ms_handle_connect 0x7f804806cac0 2024-03-29T11:45:42.443 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.427+0000 7f8036ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:42.443 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.427+0000 7f8036ffd640 10 client.?.objecter ms_handle_connect 0x7f8048145e80 2024-03-29T11:45:42.443 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.427+0000 7f8036ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:42.443 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.427+0000 7f8036ffd640 1 -- 172.21.15.72:0/2444889081 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2452019220 0 0) 0x7f80480ac220 con 0x7f804806cea0 2024-03-29T11:45:42.443 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.427+0000 7f8036ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:45:42.443 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.427+0000 7f8036ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:42.443 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.427+0000 7f8036ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:42.443 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.427+0000 7f8036ffd640 10 monclient(hunting): my global_id is 4549 2024-03-29T11:45:42.443 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.427+0000 7f8036ffd640 1 -- 172.21.15.72:0/2444889081 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8028004eb0 con 0x7f804806cea0 2024-03-29T11:45:42.443 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.428+0000 7f8036ffd640 1 -- 172.21.15.72:0/2444889081 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1609106724 0 0) 0x7f80480acf50 con 0x7f8048145e80 2024-03-29T11:45:42.443 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.428+0000 7f8036ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:45:42.444 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.428+0000 7f8036ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:42.444 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.428+0000 7f8036ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:42.444 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.428+0000 7f8036ffd640 10 monclient(hunting): my global_id is 4628 2024-03-29T11:45:42.444 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.428+0000 7f8036ffd640 1 -- 172.21.15.72:0/2444889081 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80480ac220 con 0x7f8048145e80 2024-03-29T11:45:42.444 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.428+0000 7f8036ffd640 1 -- 172.21.15.72:0/2444889081 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4199714965 0 0) 0x7f80480abf40 con 0x7f804806cac0 2024-03-29T11:45:42.444 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.428+0000 7f8036ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:45:42.444 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.428+0000 7f8036ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:42.444 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.428+0000 7f8036ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:42.444 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.428+0000 7f8036ffd640 10 monclient(hunting): my global_id is 4605 2024-03-29T11:45:42.444 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.428+0000 7f8036ffd640 1 -- 172.21.15.72:0/2444889081 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80480acf50 con 0x7f804806cac0 2024-03-29T11:45:42.444 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.428+0000 7f8036ffd640 1 -- 172.21.15.72:0/2444889081 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4277756618 0 0) 0x7f8028004eb0 con 0x7f804806cea0 2024-03-29T11:45:42.444 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.428+0000 7f8036ffd640 1 -- 172.21.15.72:0/2444889081 --> v1:172.21.15.89:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80480abf40 con 0x7f804806cea0 2024-03-29T11:45:42.444 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.429+0000 7f8036ffd640 1 -- 172.21.15.72:0/2444889081 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f803c001d60 con 0x7f804806cea0 2024-03-29T11:45:42.444 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.429+0000 7f8036ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:42.444 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.429+0000 7f8036ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:45:42.445 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.429+0000 7f8036ffd640 10 monclient(hunting): dump: 2024-03-29T11:45:42.445 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:45:42.445 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:45:42.445 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:42.445 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:42.445 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:42.445 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:42.445 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:45:42.445 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:45:42.445 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:45:42.445 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:42.445 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.429+0000 7f8036ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:42.445 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.429+0000 7f8036ffd640 1 -- 172.21.15.72:0/2444889081 <== mon.1 v1:172.21.15.89:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2809414940 0 0) 0x7f80480abf40 con 0x7f804806cea0 2024-03-29T11:45:42.445 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.429+0000 7f8036ffd640 1 -- 172.21.15.72:0/2444889081 >> v1:172.21.15.72:6790/0 conn(0x7f8048145e80 legacy=0x7f80481e2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:42.446 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.429+0000 7f8036ffd640 1 -- 172.21.15.72:0/2444889081 >> v1:172.21.15.72:6789/0 conn(0x7f804806cac0 legacy=0x7f80480ad370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:42.446 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.429+0000 7f8036ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:45:42.446 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.429+0000 7f8036ffd640 1 monclient: found mon.b 2024-03-29T11:45:42.446 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.429+0000 7f8036ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:42.446 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.429+0000 7f8036ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:45:42.446 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.429+0000 7f8036ffd640 1 -- 172.21.15.72:0/2444889081 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80481426e0 con 0x7f804806cea0 2024-03-29T11:45:42.446 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.429+0000 7f8036ffd640 10 monclient: _finish_auth 0 2024-03-29T11:45:42.446 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.429+0000 7f8036ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:45:42.446 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.429+0000 7f8036ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:42.446 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.429+0000 7f804cdc6640 5 monclient: authenticate success, global_id 4549 2024-03-29T11:45:42.446 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.429+0000 7f804cdc6640 10 monclient: _renew_subs 2024-03-29T11:45:42.446 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.429+0000 7f804cdc6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:45:42.446 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.429+0000 7f804cdc6640 1 -- 172.21.15.72:0/2444889081 --> v1:172.21.15.89:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80480abb80 con 0x7f804806cea0 2024-03-29T11:45:42.446 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.429+0000 7f804cdc6640 10 client.4549.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:42.446 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.429+0000 7f804cdc6640 10 monclient: _renew_subs 2024-03-29T11:45:42.447 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.429+0000 7f804cdc6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:45:42.447 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.429+0000 7f804cdc6640 1 -- 172.21.15.72:0/2444889081 --> v1:172.21.15.89:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f80480ab3c0 con 0x7f804806cea0 2024-03-29T11:45:42.447 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.429+0000 7f804cdc6640 1 librados: init done 2024-03-29T11:45:42.447 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.429+0000 7f8036ffd640 1 -- 172.21.15.72:0/2444889081 <== mon.1 v1:172.21.15.89:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f803c004a90 con 0x7f804806cea0 2024-03-29T11:45:42.447 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.429+0000 7f8036ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:42.447 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.429+0000 7f8036ffd640 1 -- 172.21.15.72:0/2444889081 <== mon.1 v1:172.21.15.89:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f803c004ee0 con 0x7f804806cea0 2024-03-29T11:45:42.447 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.430+0000 7f8036ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:42.447 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.430+0000 7f8036ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:45:42.447 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.430+0000 7f8036ffd640 10 monclient: dump: 2024-03-29T11:45:42.447 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:45:42.447 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:45:42.447 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:42.447 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:42.447 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:42.448 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:42.448 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:45:42.448 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:45:42.448 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:45:42.448 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:42.448 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.430+0000 7f8044d99640 4 set_mon_vals no callback set 2024-03-29T11:45:42.448 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.431+0000 7f804cdc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:42.448 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.431+0000 7f804cdc6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:42.448 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.431+0000 7f804cdc6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:45:42.448 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.431+0000 7f804cdc6640 1 -- 172.21.15.72:0/2444889081 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f802c005080 con 0x7f804806cea0 2024-03-29T11:45:42.448 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.432+0000 7f8036ffd640 1 -- 172.21.15.72:0/2444889081 <== mon.1 v1:172.21.15.89:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f803c01aeb0 con 0x7f804806cea0 2024-03-29T11:45:42.448 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.433+0000 7f804659c640 10 monclient: get_auth_request con 0x7f802806dbe0 auth_method 0 2024-03-29T11:45:42.448 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.433+0000 7f8036ffd640 1 -- 172.21.15.72:0/2444889081 <== mon.1 v1:172.21.15.89:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3662182158 0 0) 0x7f803c084c10 con 0x7f804806cea0 2024-03-29T11:45:42.448 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.433+0000 7f8036ffd640 10 client.4549.objecter ms_dispatch 0x7f8048000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:45:42.448 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.433+0000 7f8036ffd640 3 client.4549.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:45:42.448 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.433+0000 7f8036ffd640 3 client.4549.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:45:42.449 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.433+0000 7f8036ffd640 20 client.4549.objecter dump_active .. 0 homeless 2024-03-29T11:45:42.449 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.433+0000 7f8036ffd640 10 client.4549.objecter ms_handle_connect 0x7f802806dbe0 2024-03-29T11:45:42.449 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.440+0000 7f8036ffd640 1 -- 172.21.15.72:0/2444889081 <== mon.1 v1:172.21.15.89:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f803c0504f0 con 0x7f804806cea0 2024-03-29T11:45:42.449 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.440+0000 7f8036ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:42.449 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.440+0000 7f8036ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:42.842 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.837+0000 7f804cdc6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:42.842 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.837+0000 7f804cdc6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:42.842 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.837+0000 7f804cdc6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.89:6789/0 2024-03-29T11:45:42.842 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.837+0000 7f804cdc6640 1 -- 172.21.15.72:0/2444889081 --> v1:172.21.15.89:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f802c005330 con 0x7f804806cea0 2024-03-29T11:45:42.843 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.838+0000 7f8036ffd640 1 -- 172.21.15.72:0/2444889081 <== mon.1 v1:172.21.15.89:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3214350691) 0x7f803c052420 con 0x7f804806cea0 2024-03-29T11:45:42.843 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.838+0000 7f8036ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:42.843 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.838+0000 7f8036ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:42.843 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:45:42.844 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":11,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:45:05.043721+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:45:05.043721+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:42.848 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.843+0000 7f804cdc6640 10 librados: watch_flush enter 2024-03-29T11:45:42.848 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.843+0000 7f804cdc6640 10 librados: watch_flush exit 2024-03-29T11:45:42.848 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.843+0000 7f804cdc6640 20 client.4549.objecter shutdown clearing up homeless session... 2024-03-29T11:45:42.848 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.843+0000 7f804cdc6640 10 client.4549.objecter successfully canceled tick 2024-03-29T11:45:42.848 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.843+0000 7f804cdc6640 1 -- 172.21.15.72:0/2444889081 >> v1:172.21.15.72:6816/1578866102 conn(0x7f802806dbe0 legacy=0x7f80280700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:42.848 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.843+0000 7f804cdc6640 10 monclient: shutdown 2024-03-29T11:45:42.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.843+0000 7f804cdc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:42.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.843+0000 7f804cdc6640 1 -- 172.21.15.72:0/2444889081 >> v1:172.21.15.89:6789/0 conn(0x7f804806cea0 legacy=0x7f80480aaad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:42.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.843+0000 7f804cdc6640 1 -- 172.21.15.72:0/2444889081 shutdown_connections 2024-03-29T11:45:42.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.843+0000 7f804cdc6640 1 -- 172.21.15.72:0/2444889081 >> 172.21.15.72:0/2444889081 conn(0x7f80480675b0 msgr2=0x7f80480b5230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:42.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.844+0000 7f804cdc6640 1 -- 172.21.15.72:0/2444889081 shutdown_connections 2024-03-29T11:45:42.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.844+0000 7f804cdc6640 1 -- 172.21.15.72:0/2444889081 wait complete. 2024-03-29T11:45:42.849 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:42.844+0000 7f804cdc6640 1 librados: shutdown 2024-03-29T11:45:42.863 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:49.670 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:49.826 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.821+0000 7ff4c3582640 10 monclient: get_monmap_and_config 2024-03-29T11:45:49.826 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.821+0000 7ff4c3582640 10 monclient: build_initial_monmap 2024-03-29T11:45:49.826 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.821+0000 7ff4c3582640 10 monclient: monmap: 2024-03-29T11:45:49.826 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:45:49.826 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:45:49.826 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:45:49.822471+0000 2024-03-29T11:45:49.826 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:45:49.822471+0000 2024-03-29T11:45:49.826 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:49.826 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:49.826 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:45:49.826 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:45:49.827 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:45:49.827 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:49.827 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.823+0000 7ff4c3582640 1 Processor -- start 2024-03-29T11:45:49.832 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.824+0000 7ff4c3582640 1 -- start start 2024-03-29T11:45:49.832 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.824+0000 7ff4c3582640 10 monclient: init 2024-03-29T11:45:49.832 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.825+0000 7ff4c3582640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:49.832 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.825+0000 7ff4c3582640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:45:49.833 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.825+0000 7ff4c3582640 10 monclient(hunting): picked mon.noname-c con 0x7ff4b4006e50 addr v1:172.21.15.89:6789/0 2024-03-29T11:45:49.833 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.825+0000 7ff4c3582640 10 monclient(hunting): picked mon.noname-a con 0x7ff4b4035e20 addr v1:172.21.15.72:6789/0 2024-03-29T11:45:49.833 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.825+0000 7ff4c3582640 10 monclient(hunting): picked mon.noname-b con 0x7ff4b40c9690 addr v1:172.21.15.72:6790/0 2024-03-29T11:45:49.833 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.825+0000 7ff4c3582640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b4035990 con 0x7ff4b4035e20 2024-03-29T11:45:49.833 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.825+0000 7ff4c3582640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b40362a0 con 0x7ff4b4006e50 2024-03-29T11:45:49.833 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.825+0000 7ff4c3582640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b4035660 con 0x7ff4b40c9690 2024-03-29T11:45:49.833 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.825+0000 7ff4c3582640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:49.833 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.825+0000 7ff4c3582640 10 monclient(hunting): authenticate will time out at 1509.113605s 2024-03-29T11:45:49.833 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.828+0000 7ff4c1b1f640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7ff4b40c9690 0x7ff4b40d7580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:36615/0 (socket says 172.21.15.72:36615) 2024-03-29T11:45:49.833 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.828+0000 7ff4c1b1f640 1 -- 172.21.15.72:0/562777631 learned_addr learned my addr 172.21.15.72:0/562777631 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:45:49.836 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.832+0000 7ff4b3fff640 1 -- 172.21.15.72:0/562777631 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1139071397 0 0) 0x7ff4b4035990 con 0x7ff4b4035e20 2024-03-29T11:45:49.837 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.832+0000 7ff4b3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:45:49.837 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.832+0000 7ff4b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:49.837 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.832+0000 7ff4b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:49.837 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.832+0000 7ff4b3fff640 10 monclient(hunting): my global_id is 4608 2024-03-29T11:45:49.837 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.832+0000 7ff4b3fff640 1 -- 172.21.15.72:0/562777631 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4a8003520 con 0x7ff4b4035e20 2024-03-29T11:45:49.837 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.832+0000 7ff4b3fff640 1 -- 172.21.15.72:0/562777631 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2447642956 0 0) 0x7ff4a8003520 con 0x7ff4b4035e20 2024-03-29T11:45:49.837 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.832+0000 7ff4b3fff640 1 -- 172.21.15.72:0/562777631 >> v1:172.21.15.72:6790/0 conn(0x7ff4b40c9690 legacy=0x7ff4b40d7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:49.837 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.832+0000 7ff4b3fff640 1 -- 172.21.15.72:0/562777631 >> v1:172.21.15.89:6789/0 conn(0x7ff4b4006e50 legacy=0x7ff4b40cbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:49.837 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.832+0000 7ff4b3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:45:49.837 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.832+0000 7ff4b3fff640 1 monclient: found mon.noname-a 2024-03-29T11:45:49.838 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.832+0000 7ff4b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:49.838 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.832+0000 7ff4b3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.72:6789/0 2024-03-29T11:45:49.838 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.832+0000 7ff4b3fff640 1 -- 172.21.15.72:0/562777631 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4b4034e40 con 0x7ff4b4035e20 2024-03-29T11:45:49.838 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.832+0000 7ff4b3fff640 10 monclient: _finish_auth 0 2024-03-29T11:45:49.838 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.832+0000 7ff4b3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:45:49.838 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.832+0000 7ff4b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:49.838 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.832+0000 7ff4c3582640 5 monclient: authenticate success, global_id 4608 2024-03-29T11:45:49.838 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.832+0000 7ff4c3582640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:49.838 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.832+0000 7ff4b3fff640 1 -- 172.21.15.72:0/562777631 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7ff4b8001690 con 0x7ff4b4035e20 2024-03-29T11:45:49.838 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.832+0000 7ff4b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:49.838 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.832+0000 7ff4b3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:45:49.838 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.832+0000 7ff4b3fff640 10 monclient: dump: 2024-03-29T11:45:49.838 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:45:49.838 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:45:49.839 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:49.839 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:49.839 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:49.839 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:49.839 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:45:49.839 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:45:49.839 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:45:49.839 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:49.839 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.832+0000 7ff4c3582640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:49.839 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.833+0000 7ff4b3fff640 1 -- 172.21.15.72:0/562777631 <== mon.0 v1:172.21.15.72:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4b8001920 con 0x7ff4b4035e20 2024-03-29T11:45:49.839 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.833+0000 7ff4b3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:49.840 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.833+0000 7ff4c3582640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:49.840 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.833+0000 7ff4c3582640 4 set_mon_vals no callback set 2024-03-29T11:45:49.840 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.834+0000 7ff4c3582640 10 monclient: shutdown 2024-03-29T11:45:49.840 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.834+0000 7ff4c3582640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:49.840 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.834+0000 7ff4c3582640 1 -- 172.21.15.72:0/562777631 >> v1:172.21.15.72:6789/0 conn(0x7ff4b4035e20 legacy=0x7ff4b40c8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:49.841 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.836+0000 7ff4b3fff640 1 -- 172.21.15.72:0/562777631 <== mon.0 v1:172.21.15.72:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7ff4b8001690 con 0x7ff4b4035e20 2024-03-29T11:45:49.841 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.836+0000 7ff4b3fff640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:45:49.841 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.837+0000 7ff4c3582640 1 -- 172.21.15.72:0/562777631 shutdown_connections 2024-03-29T11:45:49.841 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.837+0000 7ff4c3582640 1 -- 172.21.15.72:0/562777631 >> 172.21.15.72:0/562777631 conn(0x7ff4b4005c20 msgr2=0x7ff4b4006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:49.842 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.837+0000 7ff4c3582640 1 -- 172.21.15.72:0/562777631 shutdown_connections 2024-03-29T11:45:49.845 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.840+0000 7ff4c3582640 1 -- 172.21.15.72:0/562777631 wait complete. 2024-03-29T11:45:49.846 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.841+0000 7ff4c3582640 10 monclient: build_initial_monmap 2024-03-29T11:45:49.846 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.841+0000 7ff4c3582640 10 monclient: monmap: 2024-03-29T11:45:49.846 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:45:49.847 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:49.847 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:45:49.847 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:45:49.847 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:49.847 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:49.847 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:45:49.847 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:45:49.847 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:45:49.847 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:49.847 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.842+0000 7ff4c3582640 1 librados: starting msgr at 2024-03-29T11:45:49.847 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.842+0000 7ff4c3582640 1 librados: starting objecter 2024-03-29T11:45:49.847 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.842+0000 7ff4c3582640 1 Processor -- start 2024-03-29T11:45:49.848 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.842+0000 7ff4c3582640 1 -- start start 2024-03-29T11:45:49.848 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.842+0000 7ff4c3582640 1 librados: setting wanted keys 2024-03-29T11:45:49.850 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.842+0000 7ff4c3582640 1 librados: calling monclient init 2024-03-29T11:45:49.851 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.842+0000 7ff4c3582640 10 monclient: init 2024-03-29T11:45:49.851 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.846+0000 7ff4c3582640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:49.851 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.846+0000 7ff4c3582640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:45:49.851 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.846+0000 7ff4c3582640 10 monclient(hunting): picked mon.noname-a con 0x7ff4b4006e50 addr v1:172.21.15.72:6789/0 2024-03-29T11:45:49.852 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.846+0000 7ff4c3582640 10 monclient(hunting): picked mon.noname-c con 0x7ff4b4035e20 addr v1:172.21.15.72:6790/0 2024-03-29T11:45:49.852 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.846+0000 7ff4c3582640 10 monclient(hunting): picked mon.noname-b con 0x7ff4b40c9690 addr v1:172.21.15.89:6789/0 2024-03-29T11:45:49.852 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.846+0000 7ff4c3582640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b40ccfe0 con 0x7ff4b4006e50 2024-03-29T11:45:49.852 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.846+0000 7ff4c3582640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b40cd2c0 con 0x7ff4b40c9690 2024-03-29T11:45:49.852 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.846+0000 7ff4c3582640 1 -- --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4b40cd5a0 con 0x7ff4b4035e20 2024-03-29T11:45:49.852 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.846+0000 7ff4c3582640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:49.852 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.846+0000 7ff4c3582640 10 monclient(hunting): authenticate will time out at 1509.134501s 2024-03-29T11:45:49.855 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.850+0000 7ff4c131e640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7ff4b4006e50 0x7ff4b40cd9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:36639/0 (socket says 172.21.15.72:36639) 2024-03-29T11:45:49.855 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.850+0000 7ff4c0b1d640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7ff4b4035e20 0x7ff4b4168ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:36753/0 (socket says 172.21.15.72:36753) 2024-03-29T11:45:49.855 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.850+0000 7ff4c131e640 1 -- 172.21.15.72:0/849253807 learned_addr learned my addr 172.21.15.72:0/849253807 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:45:49.855 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.850+0000 7ff4b1ffb640 10 client.?.objecter ms_handle_connect 0x7ff4b4035e20 2024-03-29T11:45:49.855 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.850+0000 7ff4b1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:49.855 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.851+0000 7ff4b1ffb640 10 client.?.objecter ms_handle_connect 0x7ff4b4006e50 2024-03-29T11:45:49.859 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.851+0000 7ff4b1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:49.859 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.851+0000 7ff4b1ffb640 1 -- 172.21.15.72:0/849253807 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2241888894 0 0) 0x7ff4b40cd5a0 con 0x7ff4b4035e20 2024-03-29T11:45:49.859 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.851+0000 7ff4b1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:45:49.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.851+0000 7ff4b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:49.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.851+0000 7ff4b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:49.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.851+0000 7ff4b1ffb640 10 monclient(hunting): my global_id is 4634 2024-03-29T11:45:49.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.851+0000 7ff4b1ffb640 1 -- 172.21.15.72:0/849253807 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4a0004eb0 con 0x7ff4b4035e20 2024-03-29T11:45:49.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.851+0000 7ff4b1ffb640 1 -- 172.21.15.72:0/849253807 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1263959452 0 0) 0x7ff4b40ccfe0 con 0x7ff4b4006e50 2024-03-29T11:45:49.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.851+0000 7ff4b1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:45:49.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.851+0000 7ff4b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:49.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.851+0000 7ff4b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:49.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.851+0000 7ff4b1ffb640 10 monclient(hunting): my global_id is 4611 2024-03-29T11:45:49.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.851+0000 7ff4b1ffb640 1 -- 172.21.15.72:0/849253807 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4b40cd5a0 con 0x7ff4b4006e50 2024-03-29T11:45:49.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.852+0000 7ff4b1ffb640 1 -- 172.21.15.72:0/849253807 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1195475623 0 0) 0x7ff4a0004eb0 con 0x7ff4b4035e20 2024-03-29T11:45:49.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.852+0000 7ff4b1ffb640 1 -- 172.21.15.72:0/849253807 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4b40ccfe0 con 0x7ff4b4035e20 2024-03-29T11:45:49.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.852+0000 7ff4b1ffb640 1 -- 172.21.15.72:0/849253807 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7ff4b8001c30 con 0x7ff4b4035e20 2024-03-29T11:45:49.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.852+0000 7ff4b1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:49.860 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.852+0000 7ff4b1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:45:49.861 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.852+0000 7ff4b1ffb640 10 monclient(hunting): dump: 2024-03-29T11:45:49.861 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:45:49.861 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:45:49.861 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:49.861 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:49.861 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:49.861 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:49.861 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:45:49.861 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:45:49.861 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:45:49.861 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:49.861 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.852+0000 7ff4b1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:49.861 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.852+0000 7ff4b1ffb640 10 client.?.objecter ms_handle_connect 0x7ff4b40c9690 2024-03-29T11:45:49.861 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.852+0000 7ff4b1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:49.861 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.852+0000 7ff4b1ffb640 1 -- 172.21.15.72:0/849253807 <== mon.2 v1:172.21.15.72:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1285939220 0 0) 0x7ff4b40ccfe0 con 0x7ff4b4035e20 2024-03-29T11:45:49.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.852+0000 7ff4b1ffb640 1 -- 172.21.15.72:0/849253807 >> v1:172.21.15.89:6789/0 conn(0x7ff4b40c9690 legacy=0x7ff4b416c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:49.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.852+0000 7ff4b1ffb640 1 -- 172.21.15.72:0/849253807 >> v1:172.21.15.72:6789/0 conn(0x7ff4b4006e50 legacy=0x7ff4b40cd9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:49.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.852+0000 7ff4b1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:45:49.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.852+0000 7ff4b1ffb640 1 monclient: found mon.c 2024-03-29T11:45:49.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.852+0000 7ff4b1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:49.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.852+0000 7ff4b1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:45:49.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.852+0000 7ff4b1ffb640 1 -- 172.21.15.72:0/849253807 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4b402e1a0 con 0x7ff4b4035e20 2024-03-29T11:45:49.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.852+0000 7ff4b1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:45:49.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.852+0000 7ff4b1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:45:49.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.852+0000 7ff4b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:49.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.852+0000 7ff4c3582640 5 monclient: authenticate success, global_id 4634 2024-03-29T11:45:49.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.852+0000 7ff4c3582640 10 monclient: _renew_subs 2024-03-29T11:45:49.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.852+0000 7ff4c3582640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:45:49.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.853+0000 7ff4c3582640 1 -- 172.21.15.72:0/849253807 --> v1:172.21.15.72:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4b40ce0d0 con 0x7ff4b4035e20 2024-03-29T11:45:49.862 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.853+0000 7ff4c3582640 10 client.4634.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:49.863 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.853+0000 7ff4c3582640 10 monclient: _renew_subs 2024-03-29T11:45:49.863 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.853+0000 7ff4c3582640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:45:49.863 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.853+0000 7ff4c3582640 1 -- 172.21.15.72:0/849253807 --> v1:172.21.15.72:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4b402f6d0 con 0x7ff4b4035e20 2024-03-29T11:45:49.863 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.853+0000 7ff4c3582640 1 librados: init done 2024-03-29T11:45:49.863 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.854+0000 7ff4b1ffb640 1 -- 172.21.15.72:0/849253807 <== mon.2 v1:172.21.15.72:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4b80038a0 con 0x7ff4b4035e20 2024-03-29T11:45:49.863 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.854+0000 7ff4b1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:49.864 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.855+0000 7ff4b2ffd640 4 set_mon_vals no callback set 2024-03-29T11:45:49.864 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.857+0000 7ff4b1ffb640 1 -- 172.21.15.72:0/849253807 <== mon.2 v1:172.21.15.72:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7ff4b8001250 con 0x7ff4b4035e20 2024-03-29T11:45:49.864 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.857+0000 7ff4b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:49.864 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.857+0000 7ff4b1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:49.864 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.857+0000 7ff4b1ffb640 10 monclient: dump: 2024-03-29T11:45:49.864 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:45:49.864 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:45:49.864 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:49.865 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:49.865 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:49.865 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:49.865 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:45:49.865 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:45:49.865 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:45:49.865 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:49.865 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.857+0000 7ff4c3582640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:49.865 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.857+0000 7ff4c3582640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:49.865 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.857+0000 7ff4c3582640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:45:49.865 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.857+0000 7ff4c3582640 1 -- 172.21.15.72:0/849253807 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff49c005080 con 0x7ff4b4035e20 2024-03-29T11:45:49.865 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.859+0000 7ff4b1ffb640 1 -- 172.21.15.72:0/849253807 <== mon.2 v1:172.21.15.72:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7ff4b8003d30 con 0x7ff4b4035e20 2024-03-29T11:45:49.865 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.859+0000 7ff4b1ffb640 1 -- 172.21.15.72:0/849253807 <== mon.2 v1:172.21.15.72:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3662182158 0 0) 0x7ff4b8084c60 con 0x7ff4b4035e20 2024-03-29T11:45:49.865 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.860+0000 7ff4c131e640 10 monclient: get_auth_request con 0x7ff4a006db50 auth_method 0 2024-03-29T11:45:49.866 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.860+0000 7ff4b1ffb640 10 client.4634.objecter ms_dispatch 0x7ff4bc000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:45:49.866 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.860+0000 7ff4b1ffb640 3 client.4634.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:45:49.866 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.860+0000 7ff4b1ffb640 3 client.4634.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:45:49.866 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.860+0000 7ff4b1ffb640 20 client.4634.objecter dump_active .. 0 homeless 2024-03-29T11:45:49.866 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.860+0000 7ff4b1ffb640 10 client.4634.objecter ms_handle_connect 0x7ff4a006db50 2024-03-29T11:45:49.872 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.868+0000 7ff4b1ffb640 1 -- 172.21.15.72:0/849253807 <== mon.2 v1:172.21.15.72:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff4b8050370 con 0x7ff4b4035e20 2024-03-29T11:45:49.872 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.868+0000 7ff4b1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:49.872 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:49.868+0000 7ff4b1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:50.214 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:50.209+0000 7ff4c3582640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:50.214 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:50.209+0000 7ff4c3582640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:50.214 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:50.209+0000 7ff4c3582640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:45:50.214 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:50.209+0000 7ff4c3582640 1 -- 172.21.15.72:0/849253807 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff49c005330 con 0x7ff4b4035e20 2024-03-29T11:45:50.215 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:50.211+0000 7ff4b1ffb640 1 -- 172.21.15.72:0/849253807 <== mon.2 v1:172.21.15.72:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3214350691) 0x7ff4b8052470 con 0x7ff4b4035e20 2024-03-29T11:45:50.215 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:50.211+0000 7ff4b1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:50.215 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:50.211+0000 7ff4b1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:50.216 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:45:50.216 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":11,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:45:05.043721+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:45:05.043721+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:50.219 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:50.214+0000 7ff4c3582640 10 librados: watch_flush enter 2024-03-29T11:45:50.219 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:50.214+0000 7ff4c3582640 10 librados: watch_flush exit 2024-03-29T11:45:50.219 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:50.214+0000 7ff4c3582640 20 client.4634.objecter shutdown clearing up homeless session... 2024-03-29T11:45:50.219 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:50.214+0000 7ff4c3582640 10 client.4634.objecter successfully canceled tick 2024-03-29T11:45:50.219 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:50.214+0000 7ff4c3582640 1 -- 172.21.15.72:0/849253807 >> v1:172.21.15.72:6816/1578866102 conn(0x7ff4a006db50 legacy=0x7ff4a0070010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:50.219 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:50.214+0000 7ff4c3582640 10 monclient: shutdown 2024-03-29T11:45:50.219 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:50.214+0000 7ff4c3582640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:50.219 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:50.214+0000 7ff4c3582640 1 -- 172.21.15.72:0/849253807 >> v1:172.21.15.72:6790/0 conn(0x7ff4b4035e20 legacy=0x7ff4b4168ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:50.220 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:50.214+0000 7ff4c3582640 1 -- 172.21.15.72:0/849253807 shutdown_connections 2024-03-29T11:45:50.220 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:50.214+0000 7ff4c3582640 1 -- 172.21.15.72:0/849253807 >> 172.21.15.72:0/849253807 conn(0x7ff4b4005c20 msgr2=0x7ff4b403ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:50.220 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:50.214+0000 7ff4c3582640 1 -- 172.21.15.72:0/849253807 shutdown_connections 2024-03-29T11:45:50.220 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:50.214+0000 7ff4c3582640 1 -- 172.21.15.72:0/849253807 wait complete. 2024-03-29T11:45:50.220 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:50.214+0000 7ff4c3582640 1 librados: shutdown 2024-03-29T11:45:50.234 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:57.041 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:57.171 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.165+0000 7f662a72b640 10 monclient: get_monmap_and_config 2024-03-29T11:45:57.171 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.165+0000 7f662a72b640 10 monclient: build_initial_monmap 2024-03-29T11:45:57.171 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.165+0000 7f662a72b640 10 monclient: monmap: 2024-03-29T11:45:57.171 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:45:57.171 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:45:57.171 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:45:57.167290+0000 2024-03-29T11:45:57.171 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:45:57.167290+0000 2024-03-29T11:45:57.171 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:57.171 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:57.171 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:45:57.171 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-c 2024-03-29T11:45:57.171 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-b 2024-03-29T11:45:57.172 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:57.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.166+0000 7f662a72b640 1 Processor -- start 2024-03-29T11:45:57.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.166+0000 7f662a72b640 1 -- start start 2024-03-29T11:45:57.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.166+0000 7f662a72b640 10 monclient: init 2024-03-29T11:45:57.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.166+0000 7f662a72b640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:57.174 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.166+0000 7f662a72b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:45:57.175 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.166+0000 7f662a72b640 10 monclient(hunting): picked mon.noname-c con 0x7f66241422b0 addr v1:172.21.15.89:6789/0 2024-03-29T11:45:57.175 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.166+0000 7f662a72b640 10 monclient(hunting): picked mon.noname-b con 0x7f6624142690 addr v1:172.21.15.72:6790/0 2024-03-29T11:45:57.175 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.167+0000 7f66237fe640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f6624142690 0x7f66240b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:43200/0 (socket says 172.21.15.72:43200) 2024-03-29T11:45:57.175 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.167+0000 7f662a72b640 10 monclient(hunting): picked mon.noname-a con 0x7f66240b17d0 addr v1:172.21.15.72:6789/0 2024-03-29T11:45:57.175 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.167+0000 7f662a72b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6624142c80 con 0x7f66240b17d0 2024-03-29T11:45:57.175 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.167+0000 7f6623fff640 1 --1- >> v1:172.21.15.89:6789/0 conn(0x7f66241422b0 0x7f662413f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.89:6789/0 says I am v1:172.21.15.72:35042/0 (socket says 172.21.15.72:35042) 2024-03-29T11:45:57.175 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.167+0000 7f6623fff640 1 -- 172.21.15.72:0/2237191214 learned_addr learned my addr 172.21.15.72:0/2237191214 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:45:57.175 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.167+0000 7f662a72b640 1 -- --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66240aae40 con 0x7f66241422b0 2024-03-29T11:45:57.175 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.167+0000 7f662a72b640 1 -- 172.21.15.72:0/2237191214 --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f662413fbd0 con 0x7f6624142690 2024-03-29T11:45:57.175 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.167+0000 7f662a72b640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:57.175 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.167+0000 7f662a72b640 10 monclient(hunting): authenticate will time out at 1516.456615s 2024-03-29T11:45:57.175 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.167+0000 7f6622ffd640 1 -- 172.21.15.72:0/2237191214 <== mon.1 v1:172.21.15.89:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1752192859 0 0) 0x7f66240aae40 con 0x7f66241422b0 2024-03-29T11:45:57.175 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.167+0000 7f6622ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:45:57.175 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.167+0000 7f6622ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:57.175 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.167+0000 7f6622ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:57.176 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.167+0000 7f6622ffd640 10 monclient(hunting): my global_id is 4558 2024-03-29T11:45:57.176 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.167+0000 7f6622ffd640 1 -- 172.21.15.72:0/2237191214 --> v1:172.21.15.89:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f660c003520 con 0x7f66241422b0 2024-03-29T11:45:57.176 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.169+0000 7f6622ffd640 1 -- 172.21.15.72:0/2237191214 <== mon.1 v1:172.21.15.89:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 417652806 0 0) 0x7f660c003520 con 0x7f66241422b0 2024-03-29T11:45:57.176 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.169+0000 7f6622ffd640 1 -- 172.21.15.72:0/2237191214 >> v1:172.21.15.72:6790/0 conn(0x7f6624142690 legacy=0x7f66240b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:45:57.176 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.169+0000 7f6622ffd640 1 -- 172.21.15.72:0/2237191214 >> v1:172.21.15.72:6789/0 conn(0x7f66240b17d0 legacy=0x7f66240b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:45:57.177 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.170+0000 7f6622ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:45:57.177 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.170+0000 7f6622ffd640 1 monclient: found mon.noname-c 2024-03-29T11:45:57.177 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.170+0000 7f6622ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:57.177 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.170+0000 7f6622ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.89:6789/0 2024-03-29T11:45:57.177 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.170+0000 7f6622ffd640 1 -- 172.21.15.72:0/2237191214 --> v1:172.21.15.89:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f662414f920 con 0x7f66241422b0 2024-03-29T11:45:57.177 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.170+0000 7f6622ffd640 10 monclient: _finish_auth 0 2024-03-29T11:45:57.177 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.170+0000 7f6622ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:45:57.177 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.170+0000 7f6622ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:57.177 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.170+0000 7f6622ffd640 1 -- 172.21.15.72:0/2237191214 <== mon.1 v1:172.21.15.89:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f6614001690 con 0x7f66241422b0 2024-03-29T11:45:57.177 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.170+0000 7f6622ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:57.177 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.171+0000 7f6622ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:45:57.177 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.171+0000 7f6622ffd640 10 monclient: dump: 2024-03-29T11:45:57.178 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:45:57.178 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:45:57.178 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:57.178 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:57.178 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:57.178 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:57.178 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:45:57.178 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:45:57.178 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:45:57.178 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:57.178 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.171+0000 7f662a72b640 5 monclient: authenticate success, global_id 4558 2024-03-29T11:45:57.178 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.171+0000 7f662a72b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:57.178 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.171+0000 7f6622ffd640 1 -- 172.21.15.72:0/2237191214 <== mon.1 v1:172.21.15.89:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6614001860 con 0x7f66241422b0 2024-03-29T11:45:57.178 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.171+0000 7f6622ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:57.179 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.171+0000 7f6622ffd640 1 -- 172.21.15.72:0/2237191214 <== mon.1 v1:172.21.15.89:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f6614001cb0 con 0x7f66241422b0 2024-03-29T11:45:57.179 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.171+0000 7f662a72b640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:57.179 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.171+0000 7f6622ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:57.179 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.171+0000 7f662a72b640 4 set_mon_vals no callback set 2024-03-29T11:45:57.179 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.171+0000 7f6622ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:45:57.179 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.171+0000 7f6622ffd640 10 monclient: dump: 2024-03-29T11:45:57.179 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:45:57.179 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:45:57.179 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:57.179 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:57.179 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:57.179 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:57.179 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:45:57.179 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:45:57.179 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:45:57.179 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:57.181 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.172+0000 7f662a72b640 10 monclient: shutdown 2024-03-29T11:45:57.182 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.172+0000 7f662a72b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:57.182 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.172+0000 7f662a72b640 1 -- 172.21.15.72:0/2237191214 >> v1:172.21.15.89:6789/0 conn(0x7f66241422b0 legacy=0x7f662413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:57.182 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.172+0000 7f662a72b640 1 -- 172.21.15.72:0/2237191214 shutdown_connections 2024-03-29T11:45:57.182 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.172+0000 7f662a72b640 1 -- 172.21.15.72:0/2237191214 >> 172.21.15.72:0/2237191214 conn(0x7f6624066a00 msgr2=0x7f6624066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:57.182 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.172+0000 7f662a72b640 1 -- 172.21.15.72:0/2237191214 shutdown_connections 2024-03-29T11:45:57.182 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.172+0000 7f662a72b640 1 -- 172.21.15.72:0/2237191214 wait complete. 2024-03-29T11:45:57.182 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.173+0000 7f662a72b640 10 monclient: build_initial_monmap 2024-03-29T11:45:57.182 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.173+0000 7f662a72b640 10 monclient: monmap: 2024-03-29T11:45:57.182 INFO:teuthology.misc.health.smithi072.stderr:epoch 0 2024-03-29T11:45:57.182 INFO:teuthology.misc.health.smithi072.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:57.182 INFO:teuthology.misc.health.smithi072.stderr:last_changed 0.000000 2024-03-29T11:45:57.182 INFO:teuthology.misc.health.smithi072.stderr:created 0.000000 2024-03-29T11:45:57.182 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:57.183 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:57.183 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.noname-a 2024-03-29T11:45:57.183 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.noname-b 2024-03-29T11:45:57.183 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.noname-c 2024-03-29T11:45:57.183 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:57.183 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.174+0000 7f662a72b640 1 librados: starting msgr at 2024-03-29T11:45:57.183 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.174+0000 7f662a72b640 1 librados: starting objecter 2024-03-29T11:45:57.183 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.174+0000 7f662a72b640 1 Processor -- start 2024-03-29T11:45:57.183 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.174+0000 7f662a72b640 1 -- start start 2024-03-29T11:45:57.183 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.174+0000 7f662a72b640 1 librados: setting wanted keys 2024-03-29T11:45:57.183 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.174+0000 7f662a72b640 1 librados: calling monclient init 2024-03-29T11:45:57.183 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.174+0000 7f662a72b640 10 monclient: init 2024-03-29T11:45:57.183 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.175+0000 7f662a72b640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:57.183 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.175+0000 7f662a72b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:45:57.183 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.175+0000 7f662a72b640 10 monclient(hunting): picked mon.noname-c con 0x7f66240b17d0 addr v1:172.21.15.72:6790/0 2024-03-29T11:45:57.183 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.176+0000 7f6623fff640 1 --1- >> v1:172.21.15.72:6790/0 conn(0x7f66240b17d0 0x7f66240adbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.72:6790/0 says I am v1:172.21.15.72:54498/0 (socket says 172.21.15.72:54498) 2024-03-29T11:45:57.184 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.176+0000 7f6623fff640 1 -- 172.21.15.72:0/3864507566 learned_addr learned my addr 172.21.15.72:0/3864507566 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-03-29T11:45:57.188 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.176+0000 7f662a72b640 10 monclient(hunting): picked mon.noname-a con 0x7f66241422b0 addr v1:172.21.15.72:6789/0 2024-03-29T11:45:57.188 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.176+0000 7f662a72b640 10 monclient(hunting): picked mon.noname-b con 0x7f6624142690 addr v1:172.21.15.89:6789/0 2024-03-29T11:45:57.188 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.176+0000 7f662a72b640 1 -- 172.21.15.72:0/3864507566 --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66240afc80 con 0x7f66241422b0 2024-03-29T11:45:57.188 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.176+0000 7f662a72b640 1 -- 172.21.15.72:0/3864507566 --> v1:172.21.15.89:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66240aee70 con 0x7f6624142690 2024-03-29T11:45:57.188 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.176+0000 7f662a72b640 1 -- 172.21.15.72:0/3864507566 --> v1:172.21.15.72:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66240af150 con 0x7f66240b17d0 2024-03-29T11:45:57.188 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.176+0000 7f662a72b640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:57.188 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.176+0000 7f662a72b640 10 monclient(hunting): authenticate will time out at 1516.465534s 2024-03-29T11:45:57.188 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.177+0000 7f6620ff9640 10 client.?.objecter ms_handle_connect 0x7f66240b17d0 2024-03-29T11:45:57.188 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.177+0000 7f6620ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:57.188 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.177+0000 7f6620ff9640 1 -- 172.21.15.72:0/3864507566 <== mon.2 v1:172.21.15.72:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 897477981 0 0) 0x7f66240af150 con 0x7f66240b17d0 2024-03-29T11:45:57.188 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.177+0000 7f6620ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:45:57.189 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.177+0000 7f6620ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:57.189 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.177+0000 7f6620ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:57.189 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.177+0000 7f6620ff9640 10 monclient(hunting): my global_id is 4637 2024-03-29T11:45:57.189 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.177+0000 7f6620ff9640 1 -- 172.21.15.72:0/3864507566 --> v1:172.21.15.72:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6610004eb0 con 0x7f66240b17d0 2024-03-29T11:45:57.189 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.177+0000 7f6620ff9640 10 client.?.objecter ms_handle_connect 0x7f6624142690 2024-03-29T11:45:57.189 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.177+0000 7f6620ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:57.189 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.178+0000 7f6620ff9640 1 -- 172.21.15.72:0/3864507566 <== mon.2 v1:172.21.15.72:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1476598718 0 0) 0x7f6610004eb0 con 0x7f66240b17d0 2024-03-29T11:45:57.189 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.178+0000 7f6620ff9640 1 -- 172.21.15.72:0/3864507566 --> v1:172.21.15.72:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66240af150 con 0x7f66240b17d0 2024-03-29T11:45:57.189 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.178+0000 7f6620ff9640 1 -- 172.21.15.72:0/3864507566 <== mon.2 v1:172.21.15.72:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f6614001b70 con 0x7f66240b17d0 2024-03-29T11:45:57.189 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.178+0000 7f6620ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:57.189 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.178+0000 7f6620ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:45:57.189 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.178+0000 7f6620ff9640 10 monclient(hunting): dump: 2024-03-29T11:45:57.189 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:45:57.189 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:45:57.189 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:57.190 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:57.190 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:57.190 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:57.190 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:45:57.190 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:45:57.190 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:45:57.190 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:57.190 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.178+0000 7f6620ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:57.190 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.178+0000 7f6620ff9640 1 -- 172.21.15.72:0/3864507566 <== mon.2 v1:172.21.15.72:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1330126671 0 0) 0x7f66240af150 con 0x7f66240b17d0 2024-03-29T11:45:57.190 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.178+0000 7f6620ff9640 1 -- 172.21.15.72:0/3864507566 >> v1:172.21.15.89:6789/0 conn(0x7f6624142690 legacy=0x7f66241e1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:57.190 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.178+0000 7f6620ff9640 1 -- 172.21.15.72:0/3864507566 >> v1:172.21.15.72:6789/0 conn(0x7f66241422b0 legacy=0x7f66240af570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:45:57.190 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.179+0000 7f6620ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:45:57.190 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.179+0000 7f6620ff9640 1 monclient: found mon.c 2024-03-29T11:45:57.190 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.179+0000 7f6620ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:57.191 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.179+0000 7f6620ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:45:57.191 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.179+0000 7f6620ff9640 1 -- 172.21.15.72:0/3864507566 --> v1:172.21.15.72:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66240ad6a0 con 0x7f66240b17d0 2024-03-29T11:45:57.191 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.179+0000 7f6620ff9640 10 monclient: _finish_auth 0 2024-03-29T11:45:57.191 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.179+0000 7f6620ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:45:57.191 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.179+0000 7f6620ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:57.191 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.179+0000 7f662a72b640 5 monclient: authenticate success, global_id 4637 2024-03-29T11:45:57.191 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.179+0000 7f662a72b640 10 monclient: _renew_subs 2024-03-29T11:45:57.191 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.179+0000 7f662a72b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:45:57.191 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.179+0000 7f662a72b640 1 -- 172.21.15.72:0/3864507566 --> v1:172.21.15.72:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66240acaf0 con 0x7f66240b17d0 2024-03-29T11:45:57.191 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.179+0000 7f662a72b640 10 client.4637.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:57.191 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.179+0000 7f662a72b640 10 monclient: _renew_subs 2024-03-29T11:45:57.191 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.179+0000 7f662a72b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:45:57.191 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.179+0000 7f662a72b640 1 -- 172.21.15.72:0/3864507566 --> v1:172.21.15.72:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6624133ef0 con 0x7f66240b17d0 2024-03-29T11:45:57.191 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.179+0000 7f662a72b640 1 librados: init done 2024-03-29T11:45:57.191 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.179+0000 7f6620ff9640 1 -- 172.21.15.72:0/3864507566 <== mon.2 v1:172.21.15.72:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6614001cd0 con 0x7f66240b17d0 2024-03-29T11:45:57.192 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.179+0000 7f6620ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:57.192 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.179+0000 7f66227fc640 4 set_mon_vals no callback set 2024-03-29T11:45:57.192 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.179+0000 7f6620ff9640 1 -- 172.21.15.72:0/3864507566 <== mon.2 v1:172.21.15.72:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3004288456 0 0) 0x7f6614004dd0 con 0x7f66240b17d0 2024-03-29T11:45:57.192 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.179+0000 7f6620ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:57.192 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.179+0000 7f6620ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:57.192 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.179+0000 7f6620ff9640 10 monclient: dump: 2024-03-29T11:45:57.192 INFO:teuthology.misc.health.smithi072.stderr:epoch 1 2024-03-29T11:45:57.192 INFO:teuthology.misc.health.smithi072.stderr:fsid 5ccd1c04-292b-4e92-9d69-700bf1c298b0 2024-03-29T11:45:57.192 INFO:teuthology.misc.health.smithi072.stderr:last_changed 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:57.192 INFO:teuthology.misc.health.smithi072.stderr:created 2024-03-29T11:34:21.957182+0000 2024-03-29T11:45:57.192 INFO:teuthology.misc.health.smithi072.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:57.192 INFO:teuthology.misc.health.smithi072.stderr:election_strategy: 1 2024-03-29T11:45:57.192 INFO:teuthology.misc.health.smithi072.stderr:0: v1:172.21.15.72:6789/0 mon.a 2024-03-29T11:45:57.192 INFO:teuthology.misc.health.smithi072.stderr:1: v1:172.21.15.89:6789/0 mon.b 2024-03-29T11:45:57.192 INFO:teuthology.misc.health.smithi072.stderr:2: v1:172.21.15.72:6790/0 mon.c 2024-03-29T11:45:57.193 INFO:teuthology.misc.health.smithi072.stderr: 2024-03-29T11:45:57.193 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.181+0000 7f662a72b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:57.193 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.181+0000 7f662a72b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:57.193 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.181+0000 7f662a72b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:45:57.193 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.181+0000 7f662a72b640 1 -- 172.21.15.72:0/3864507566 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6604005080 con 0x7f66240b17d0 2024-03-29T11:45:57.193 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.182+0000 7f6620ff9640 1 -- 172.21.15.72:0/3864507566 <== mon.2 v1:172.21.15.72:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2476663104 0 0) 0x7f661401ae20 con 0x7f66240b17d0 2024-03-29T11:45:57.194 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.183+0000 7f66237fe640 10 monclient: get_auth_request con 0x7f661006dd60 auth_method 0 2024-03-29T11:45:57.194 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.184+0000 7f6620ff9640 1 -- 172.21.15.72:0/3864507566 <== mon.2 v1:172.21.15.72:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3662182158 0 0) 0x7f6614084b90 con 0x7f66240b17d0 2024-03-29T11:45:57.194 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.184+0000 7f6620ff9640 10 client.4637.objecter ms_dispatch 0x7f6624000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:45:57.194 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.184+0000 7f6620ff9640 3 client.4637.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:45:57.194 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.184+0000 7f6620ff9640 3 client.4637.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:45:57.194 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.184+0000 7f6620ff9640 20 client.4637.objecter dump_active .. 0 homeless 2024-03-29T11:45:57.195 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.184+0000 7f6620ff9640 10 client.4637.objecter ms_handle_connect 0x7f661006dd60 2024-03-29T11:45:57.195 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.189+0000 7f6620ff9640 1 -- 172.21.15.72:0/3864507566 <== mon.2 v1:172.21.15.72:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f66140502a0 con 0x7f66240b17d0 2024-03-29T11:45:57.195 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.189+0000 7f6620ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:57.195 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.189+0000 7f6620ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:57.497 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.491+0000 7f662a72b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:57.498 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.492+0000 7f662a72b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:57.498 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.492+0000 7f662a72b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.72:6790/0 2024-03-29T11:45:57.498 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.492+0000 7f662a72b640 1 -- 172.21.15.72:0/3864507566 --> v1:172.21.15.72:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6604005330 con 0x7f66240b17d0 2024-03-29T11:45:57.498 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.493+0000 7f6620ff9640 1 -- 172.21.15.72:0/3864507566 <== mon.2 v1:172.21.15.72:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3214350691) 0x7f66140523a0 con 0x7f66240b17d0 2024-03-29T11:45:57.498 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.493+0000 7f6620ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:57.498 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.493+0000 7f6620ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:57.499 INFO:teuthology.misc.health.smithi072.stdout: 2024-03-29T11:45:57.499 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":11,"fsid":"5ccd1c04-292b-4e92-9d69-700bf1c298b0","created":"2024-03-29T11:35:00.100224+0000","modified":"2024-03-29T11:45:05.043721+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:45:05.043721+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"a2a7cb44-331d-432e-a832-f28309f32297","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"e054ff7c-3661-44b2-b369-5dbf77516540","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"fdaa9cde-4518-4a6b-a76f-526108e0ce08","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"1497998c-9d8a-47c5-ad4d-21f1a96c41b3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fa9d435f-3f50-49a3-be51-8848398cdd8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e044742d-5adf-4444-9725-f7a9ac404cf9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7c9b4262-0624-4322-b4be-78b105d0608e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"5491b1cb-cff3-4c28-803f-bc9c40c28b35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:57.502 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.496+0000 7f662a72b640 10 librados: watch_flush enter 2024-03-29T11:45:57.502 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.496+0000 7f662a72b640 10 librados: watch_flush exit 2024-03-29T11:45:57.502 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.496+0000 7f662a72b640 20 client.4637.objecter shutdown clearing up homeless session... 2024-03-29T11:45:57.502 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.496+0000 7f662a72b640 10 client.4637.objecter successfully canceled tick 2024-03-29T11:45:57.502 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.496+0000 7f662a72b640 1 -- 172.21.15.72:0/3864507566 >> v1:172.21.15.72:6816/1578866102 conn(0x7f661006dd60 legacy=0x7f6610070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:57.502 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.496+0000 7f662a72b640 10 monclient: shutdown 2024-03-29T11:45:57.502 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.496+0000 7f662a72b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:57.502 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.496+0000 7f662a72b640 1 -- 172.21.15.72:0/3864507566 >> v1:172.21.15.72:6790/0 conn(0x7f66240b17d0 legacy=0x7f66240adbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:57.503 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.497+0000 7f662a72b640 1 -- 172.21.15.72:0/3864507566 shutdown_connections 2024-03-29T11:45:57.503 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.497+0000 7f662a72b640 1 -- 172.21.15.72:0/3864507566 >> 172.21.15.72:0/3864507566 conn(0x7f6624066a00 msgr2=0x7f662413d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:57.503 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.497+0000 7f662a72b640 1 -- 172.21.15.72:0/3864507566 shutdown_connections 2024-03-29T11:45:57.503 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.497+0000 7f662a72b640 1 -- 172.21.15.72:0/3864507566 wait complete. 2024-03-29T11:45:57.503 INFO:teuthology.misc.health.smithi072.stderr:2024-03-29T11:45:57.497+0000 7f662a72b640 1 librados: shutdown 2024-03-29T11:45:57.519 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:57.519 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:45:57.521 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T11:45:57.521 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T11:45:57.521 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:57.556 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T11:45:57.557 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T11:45:57.557 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:57.625 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T11:45:57.625 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T11:45:57.625 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:57.697 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T11:45:57.697 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T11:45:57.697 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:57.769 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T11:45:57.770 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T11:45:57.770 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:57.810 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T11:45:57.811 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T11:45:57.811 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:57.883 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T11:45:57.883 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T11:45:57.883 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:57.955 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T11:45:57.956 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T11:45:57.956 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:58.026 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T11:45:58.027 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T11:45:58.027 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T11:45:58.027 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:58.051 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T11:45:58.051 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T11:45:58.051 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:58.091 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T11:45:58.091 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T11:45:58.091 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T11:45:58.092 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:58.103 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T11:45:58.103 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T11:45:58.103 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:58.154 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T11:45:58.155 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T11:45:58.155 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:45:58.166 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T11:45:58.166 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T11:45:58.166 DEBUG:teuthology.orchestra.run.smithi072:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T11:45:58.229 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi072.front.sepia.ceph.com 2024-03-29T11:45:58.229 DEBUG:teuthology.orchestra.run.smithi072:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T11:46:07.394 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi072.front.sepia.ceph.com 2024-03-29T11:46:07.394 DEBUG:teuthology.orchestra.run.smithi072:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T11:46:07.497 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi072.front.sepia.ceph.com 2024-03-29T11:46:07.498 DEBUG:teuthology.orchestra.run.smithi072:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T11:46:07.556 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi072.front.sepia.ceph.com 2024-03-29T11:46:07.556 DEBUG:teuthology.orchestra.run.smithi072:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T11:46:07.633 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi089.front.sepia.ceph.com 2024-03-29T11:46:07.633 DEBUG:teuthology.orchestra.run.smithi089:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T11:46:10.925 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi089.front.sepia.ceph.com 2024-03-29T11:46:10.926 DEBUG:teuthology.orchestra.run.smithi089:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T11:46:11.028 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi089.front.sepia.ceph.com 2024-03-29T11:46:11.028 DEBUG:teuthology.orchestra.run.smithi089:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T11:46:11.087 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi089.front.sepia.ceph.com 2024-03-29T11:46:11.087 DEBUG:teuthology.orchestra.run.smithi089:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T11:46:11.162 INFO:tasks.ceph:Archiving mon data... 2024-03-29T11:46:11.164 DEBUG:teuthology.misc:Transferring archived files from smithi072:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622722/data/mon.a.tgz 2024-03-29T11:46:11.164 DEBUG:teuthology.orchestra.run.smithi072:> mktemp 2024-03-29T11:46:11.182 INFO:teuthology.orchestra.run.smithi072.stdout:/tmp/tmp.Oa52K5vGPQ 2024-03-29T11:46:11.183 DEBUG:teuthology.orchestra.run.smithi072:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.Oa52K5vGPQ 2024-03-29T11:46:11.306 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 0666 /tmp/tmp.Oa52K5vGPQ 2024-03-29T11:46:11.394 DEBUG:teuthology.orchestra.remote:smithi072:/tmp/tmp.Oa52K5vGPQ is 211KB 2024-03-29T11:46:11.488 DEBUG:teuthology.orchestra.run.smithi072:> rm -fr /tmp/tmp.Oa52K5vGPQ 2024-03-29T11:46:11.503 DEBUG:teuthology.misc:Transferring archived files from smithi072:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622722/data/mon.c.tgz 2024-03-29T11:46:11.503 DEBUG:teuthology.orchestra.run.smithi072:> mktemp 2024-03-29T11:46:11.558 INFO:teuthology.orchestra.run.smithi072.stdout:/tmp/tmp.T4tUulPEYH 2024-03-29T11:46:11.559 DEBUG:teuthology.orchestra.run.smithi072:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.T4tUulPEYH 2024-03-29T11:46:11.658 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 0666 /tmp/tmp.T4tUulPEYH 2024-03-29T11:46:11.742 DEBUG:teuthology.orchestra.remote:smithi072:/tmp/tmp.T4tUulPEYH is 211KB 2024-03-29T11:46:11.836 DEBUG:teuthology.orchestra.run.smithi072:> rm -fr /tmp/tmp.T4tUulPEYH 2024-03-29T11:46:11.850 DEBUG:teuthology.misc:Transferring archived files from smithi089:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622722/data/mon.b.tgz 2024-03-29T11:46:11.850 DEBUG:teuthology.orchestra.run.smithi089:> mktemp 2024-03-29T11:46:11.868 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/tmp.llMSPw7zRH 2024-03-29T11:46:11.868 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.llMSPw7zRH 2024-03-29T11:46:11.989 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 0666 /tmp/tmp.llMSPw7zRH 2024-03-29T11:46:12.076 DEBUG:teuthology.orchestra.remote:smithi089:/tmp/tmp.llMSPw7zRH is 211KB 2024-03-29T11:46:12.171 DEBUG:teuthology.orchestra.run.smithi089:> rm -fr /tmp/tmp.llMSPw7zRH 2024-03-29T11:46:12.186 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T11:46:12.187 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:46:12.189 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:46:12.269 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:46:12.309 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:46:12.310 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T11:46:12.311 DEBUG:teuthology.misc:Transferring archived files from smithi072:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622722/remote/smithi072/crash 2024-03-29T11:46:12.312 DEBUG:teuthology.orchestra.run.smithi072:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:46:12.340 DEBUG:teuthology.misc:Transferring archived files from smithi089:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622722/remote/smithi089/crash 2024-03-29T11:46:12.341 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:46:12.376 INFO:tasks.ceph:Compressing logs... 2024-03-29T11:46:12.376 DEBUG:teuthology.orchestra.run.smithi072:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:46:12.384 DEBUG:teuthology.orchestra.run.smithi089:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:46:12.408 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40158.log 2024-03-29T11:46:12.408 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40392.log 2024-03-29T11:46:12.441 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.40158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38853.log 2024-03-29T11:46:12.442 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.40392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39947.log 2024-03-29T11:46:12.442 INFO:teuthology.orchestra.run.smithi072.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40158.log.gz 2024-03-29T11:46:12.442 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38853.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T11:46:12.442 INFO:teuthology.orchestra.run.smithi072.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40392.log.gz 2024-03-29T11:46:12.442 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.39947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39243.log 2024-03-29T11:46:12.442 INFO:teuthology.orchestra.run.smithi072.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.38853.log.gz 2024-03-29T11:46:12.442 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41173.log 2024-03-29T11:46:12.442 INFO:teuthology.orchestra.run.smithi072.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39947.log.gz 2024-03-29T11:46:12.442 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.39243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38814.log 2024-03-29T11:46:12.442 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.41173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40197.log 2024-03-29T11:46:12.442 INFO:teuthology.orchestra.run.smithi072.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39243.log.gz 2024-03-29T11:46:12.442 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38970.log 2024-03-29T11:46:12.442 INFO:teuthology.orchestra.run.smithi072.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41173.log.gz 2024-03-29T11:46:12.442 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.40197.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38814.log.gz 2024-03-29T11:46:12.442 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39633.log 2024-03-29T11:46:12.443 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39360.log 2024-03-29T11:46:12.443 INFO:teuthology.orchestra.run.smithi072.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40197.log.gz 2024-03-29T11:46:12.443 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.39633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38931.log 2024-03-29T11:46:12.443 INFO:teuthology.orchestra.run.smithi072.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38970.log.gz 2024-03-29T11:46:12.443 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.39360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38187.log 2024-03-29T11:46:12.443 INFO:teuthology.orchestra.run.smithi072.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39633.log.gz 2024-03-29T11:46:12.443 INFO:teuthology.orchestra.run.smithi072.stderr: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.39360.log.gz 2024-03-29T11:46:12.443 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39399.log 2024-03-29T11:46:12.443 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40900.log 2024-03-29T11:46:12.443 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38187.log: /var/log/ceph/ceph-client.admin.39399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40665.log 2024-03-29T11:46:12.443 INFO:teuthology.orchestra.run.smithi072.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38931.log.gz 2024-03-29T11:46:12.443 INFO:teuthology.orchestra.run.smithi072.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38187.log.gz 2024-03-29T11:46:12.443 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.40900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39711.log 2024-03-29T11:46:12.443 INFO:teuthology.orchestra.run.smithi072.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39399.log.gz 2024-03-29T11:46:12.443 INFO:teuthology.orchestra.run.smithi072.stderr:gzip/var/log/ceph/ceph-client.admin.40665.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39869.log 2024-03-29T11:46:12.443 INFO:teuthology.orchestra.run.smithi072.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40900.log.gz 2024-03-29T11:46:12.443 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.39711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39555.log 2024-03-29T11:46:12.444 INFO:teuthology.orchestra.run.smithi072.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40665.log.gz 2024-03-29T11:46:12.444 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.39869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41329.log 2024-03-29T11:46:12.444 INFO:teuthology.orchestra.run.smithi072.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39711.log.gz 2024-03-29T11:46:12.444 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.39555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39908.log 2024-03-29T11:46:12.444 INFO:teuthology.orchestra.run.smithi072.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39869.log.gz 2024-03-29T11:46:12.444 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.41329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41134.log 2024-03-29T11:46:12.444 INFO:teuthology.orchestra.run.smithi072.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39555.log.gz 2024-03-29T11:46:12.444 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.39908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38501.log 2024-03-29T11:46:12.444 INFO:teuthology.orchestra.run.smithi072.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41329.log.gz 2024-03-29T11:46:12.444 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.41134.log: gzip 86.3% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.39908.log.gz -- 2024-03-29T11:46:12.444 INFO:teuthology.orchestra.run.smithi072.stderr: /var/log/ceph/ceph-client.admin.38343.log 2024-03-29T11:46:12.444 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38501.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41134.log.gz 2024-03-29T11:46:12.444 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34803.log 2024-03-29T11:46:12.444 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41602.log 2024-03-29T11:46:12.444 INFO:teuthology.orchestra.run.smithi072.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38501.log.gz 2024-03-29T11:46:12.444 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.34803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40860.log 2024-03-29T11:46:12.444 INFO:teuthology.orchestra.run.smithi072.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38343.log.gz 2024-03-29T11:46:12.445 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.41602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38579.log 2024-03-29T11:46:12.445 INFO:teuthology.orchestra.run.smithi072.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34803.log.gz 2024-03-29T11:46:12.445 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.40860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39048.log 2024-03-29T11:46:12.445 INFO:teuthology.orchestra.run.smithi072.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41602.log.gz 2024-03-29T11:46:12.445 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38226.log 2024-03-29T11:46:12.445 INFO:teuthology.orchestra.run.smithi072.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40860.log.gz 2024-03-29T11:46:12.445 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.39048.log: gzip -5 85.8% --verbose -- replaced with /var/log/ceph/ceph-client.admin.38579.log.gz -- 2024-03-29T11:46:12.445 INFO:teuthology.orchestra.run.smithi072.stderr: /var/log/ceph/ceph-client.admin.38697.log 2024-03-29T11:46:12.445 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41641.log 2024-03-29T11:46:12.445 INFO:teuthology.orchestra.run.smithi072.stderr: 85.2% -- replaced with /var/log/ceph/ceph-client.admin.39048.log.gz 2024-03-29T11:46:12.445 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41017.log 2024-03-29T11:46:12.445 INFO:teuthology.orchestra.run.smithi072.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38226.log.gz 2024-03-29T11:46:12.445 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.41641.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38697.log.gz 2024-03-29T11:46:12.445 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39830.log 2024-03-29T11:46:12.445 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.41017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39438.log 2024-03-29T11:46:12.445 INFO:teuthology.orchestra.run.smithi072.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41641.log.gz 2024-03-29T11:46:12.446 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.39830.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 85.6% 2024-03-29T11:46:12.446 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41017.log.gz 2024-03-29T11:46:12.446 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40548.log 2024-03-29T11:46:12.446 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.39438.log: /var/log/ceph/ceph.audit.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39830.log.gz 2024-03-29T11:46:12.446 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41524.log 2024-03-29T11:46:12.446 INFO:teuthology.orchestra.run.smithi072.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39438.log.gz 2024-03-29T11:46:12.446 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.40548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34512.log 2024-03-29T11:46:12.446 INFO:teuthology.orchestra.run.smithi072.stderr: 93.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:46:12.446 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38775.log 2024-03-29T11:46:12.446 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.41524.log: /var/log/ceph/ceph-client.admin.34512.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40548.log.gz 2024-03-29T11:46:12.446 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38736.log 2024-03-29T11:46:12.446 INFO:teuthology.orchestra.run.smithi072.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41524.log.gz 2024-03-29T11:46:12.446 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41056.log 2024-03-29T11:46:12.446 INFO:teuthology.orchestra.run.smithi072.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.34512.log.gz 2024-03-29T11:46:12.446 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38304.log 2024-03-29T11:46:12.446 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38736.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38775.log.gz 2024-03-29T11:46:12.447 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.41056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39986.log 2024-03-29T11:46:12.447 INFO:teuthology.orchestra.run.smithi072.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38736.log.gz 2024-03-29T11:46:12.447 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40939.log 2024-03-29T11:46:12.447 INFO:teuthology.orchestra.run.smithi072.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41056.log.gz 2024-03-29T11:46:12.447 INFO:teuthology.orchestra.run.smithi072.stderr:gzip 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38304.log.gz/var/log/ceph/ceph-client.admin.39986.log: -5 2024-03-29T11:46:12.447 INFO:teuthology.orchestra.run.smithi072.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41368.log 2024-03-29T11:46:12.447 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.40939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41212.log 2024-03-29T11:46:12.447 INFO:teuthology.orchestra.run.smithi072.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39986.log.gz 2024-03-29T11:46:12.447 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.41368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41446.log 2024-03-29T11:46:12.447 INFO:teuthology.orchestra.run.smithi072.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40939.log.gz 2024-03-29T11:46:12.447 INFO:teuthology.orchestra.run.smithi072.stderr: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.41368.log.gz 2024-03-29T11:46:12.447 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T11:46:12.447 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.41212.log: /var/log/ceph/ceph-client.admin.41446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38619.log 2024-03-29T11:46:12.447 INFO:teuthology.orchestra.run.smithi072.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41212.log.gz 2024-03-29T11:46:12.447 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40782.log 2024-03-29T11:46:12.448 INFO:teuthology.orchestra.run.smithi072.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41446.log.gz 2024-03-29T11:46:12.448 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40025.log 2024-03-29T11:46:12.448 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38619.log: /var/log/ceph/ceph-client.admin.40782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40314.log 2024-03-29T11:46:12.448 INFO:teuthology.orchestra.run.smithi072.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38619.log.gz 2024-03-29T11:46:12.448 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39672.log 2024-03-29T11:46:12.448 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.40025.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40782.log.gz 2024-03-29T11:46:12.448 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.40314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40275.log 2024-03-29T11:46:12.448 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.39672.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40025.log.gz 2024-03-29T11:46:12.448 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39789.log 2024-03-29T11:46:12.448 INFO:teuthology.orchestra.run.smithi072.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40314.log.gz 2024-03-29T11:46:12.448 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.40275.log: 85.2% -- replaced with /var/log/ceph/ceph-client.admin.39672.log.gz 2024-03-29T11:46:12.448 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39087.log 2024-03-29T11:46:12.448 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39204.log 2024-03-29T11:46:12.448 INFO:teuthology.orchestra.run.smithi072.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40275.log.gz 2024-03-29T11:46:12.448 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.39789.log: /var/log/ceph/ceph-client.admin.39087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34906.log 2024-03-29T11:46:12.449 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.39204.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39789.log.gz 2024-03-29T11:46:12.449 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40704.log 2024-03-29T11:46:12.449 INFO:teuthology.orchestra.run.smithi072.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39087.log.gz 2024-03-29T11:46:12.449 INFO:teuthology.orchestra.run.smithi072.stderr:gzip/var/log/ceph/ceph-client.admin.34906.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41485.log 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39204.log.gz 2024-03-29T11:46:12.449 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-03-29T11:46:12.449 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.40704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40626.log 2024-03-29T11:46:12.449 INFO:teuthology.orchestra.run.smithi072.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34906.log.gz 2024-03-29T11:46:12.449 INFO:teuthology.orchestra.run.smithi072.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40704.log.gz 2024-03-29T11:46:12.449 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.41485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38265.log 2024-03-29T11:46:12.449 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.40626.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T11:46:12.449 INFO:teuthology.orchestra.run.smithi072.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41485.log.gz 2024-03-29T11:46:12.449 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40078.log 2024-03-29T11:46:12.449 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38265.log: /var/log/ceph/ceph-osd.3.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40626.log.gz 2024-03-29T11:46:12.449 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40470.log 2024-03-29T11:46:12.449 INFO:teuthology.orchestra.run.smithi072.stderr: 85.2% -- replaced with /var/log/ceph/ceph-client.admin.38265.log.gz 2024-03-29T11:46:12.450 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.40078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40236.log 2024-03-29T11:46:12.450 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.40470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41563.log 2024-03-29T11:46:12.450 INFO:teuthology.orchestra.run.smithi072.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40078.log.gz 2024-03-29T11:46:12.450 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.40236.log: 86.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.40470.log.gz -5 2024-03-29T11:46:12.450 INFO:teuthology.orchestra.run.smithi072.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39750.log 2024-03-29T11:46:12.450 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40431.log 2024-03-29T11:46:12.450 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.41563.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40236.log.gz 2024-03-29T11:46:12.450 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.39750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39516.log 2024-03-29T11:46:12.450 INFO:teuthology.orchestra.run.smithi072.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41563.log.gz 2024-03-29T11:46:12.450 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.40431.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39750.log.gz 2024-03-29T11:46:12.450 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40743.log 2024-03-29T11:46:12.450 INFO:teuthology.orchestra.run.smithi072.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40431.log.gz 2024-03-29T11:46:12.450 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.39516.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T11:46:12.450 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.40743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41095.log 2024-03-29T11:46:12.450 INFO:teuthology.orchestra.run.smithi072.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39516.log.gz 2024-03-29T11:46:12.451 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38382.log 2024-03-29T11:46:12.451 INFO:teuthology.orchestra.run.smithi072.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40743.log.gz 2024-03-29T11:46:12.451 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.41095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40587.log 2024-03-29T11:46:12.451 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38540.log 2024-03-29T11:46:12.451 INFO:teuthology.orchestra.run.smithi072.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41095.log.gz 2024-03-29T11:46:12.451 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.40587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39477.log 2024-03-29T11:46:12.451 INFO:teuthology.orchestra.run.smithi072.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38382.log.gzgzip 2024-03-29T11:46:12.451 INFO:teuthology.orchestra.run.smithi072.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40117.log 2024-03-29T11:46:12.451 INFO:teuthology.orchestra.run.smithi072.stderr: 86.8%/var/log/ceph/ceph-client.admin.39477.log: /var/log/ceph/ceph-client.admin.38540.log: -- replaced with /var/log/ceph/ceph-client.admin.40587.log.gz 2024-03-29T11:46:12.451 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38421.log 2024-03-29T11:46:12.451 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.40117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41251.log 2024-03-29T11:46:12.451 INFO:teuthology.orchestra.run.smithi072.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39477.log.gz 2024-03-29T11:46:12.451 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38421.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38540.log.gz 2024-03-29T11:46:12.451 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40117.log.gz 2024-03-29T11:46:12.451 INFO:teuthology.orchestra.run.smithi072.stderr: -- /var/log/ceph/ceph-client.admin.39126.log 2024-03-29T11:46:12.452 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.41251.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38421.log.gz 2024-03-29T11:46:12.452 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39321.log 2024-03-29T11:46:12.452 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.39126.log: 86.9%gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-29T11:46:12.452 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41251.log.gz 2024-03-29T11:46:12.452 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.39321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40978.log 2024-03-29T11:46:12.452 INFO:teuthology.orchestra.run.smithi072.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39126.log.gz 2024-03-29T11:46:12.452 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39009.log 2024-03-29T11:46:12.452 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-osd.0.log: 87.4%/var/log/ceph/ceph-client.admin.40978.log: -- replaced with /var/log/ceph/ceph-client.admin.39321.log.gz 2024-03-29T11:46:12.452 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39594.log 2024-03-29T11:46:12.452 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.39009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38462.log 2024-03-29T11:46:12.452 INFO:teuthology.orchestra.run.smithi072.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40978.log.gz 2024-03-29T11:46:12.452 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.39594.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39009.log.gz 2024-03-29T11:46:12.452 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40353.log 2024-03-29T11:46:12.452 INFO:teuthology.orchestra.run.smithi072.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39594.log.gz 2024-03-29T11:46:12.452 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38462.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38462.log.gz 2024-03-29T11:46:12.453 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:46:12.453 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.40353.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T11:46:12.453 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph.log: 85.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38892.log 2024-03-29T11:46:12.453 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40353.log.gz 2024-03-29T11:46:12.453 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-osd.2.log: 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:46:12.463 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39165.log 2024-03-29T11:46:12.463 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41290.log 2024-03-29T11:46:12.463 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.39165.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38892.log.gz 2024-03-29T11:46:12.463 INFO:teuthology.orchestra.run.smithi072.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39165.log.gz 2024-03-29T11:46:12.464 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39282.log 2024-03-29T11:46:12.464 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.41290.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28693.log 2024-03-29T11:46:12.464 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.39282.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41290.log.gz 2024-03-29T11:46:12.464 INFO:teuthology.orchestra.run.smithi072.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39282.log.gz 2024-03-29T11:46:12.464 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38658.log 2024-03-29T11:46:12.464 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph.tmp-client.admin.28693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40509.log 2024-03-29T11:46:12.464 INFO:teuthology.orchestra.run.smithi072.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28693.log.gz 2024-03-29T11:46:12.464 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.38658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41407.log 2024-03-29T11:46:12.464 INFO:teuthology.orchestra.run.smithi072.stderr: 85.8%/var/log/ceph/ceph-client.admin.40509.log: -- replaced with /var/log/ceph/ceph-client.admin.38658.log.gz 2024-03-29T11:46:12.464 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40509.log.gz 2024-03-29T11:46:12.464 INFO:teuthology.orchestra.run.smithi072.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40821.log 2024-03-29T11:46:12.464 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.41407.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T11:46:12.464 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.40821.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41407.log.gz 2024-03-29T11:46:12.464 INFO:teuthology.orchestra.run.smithi072.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.40821.log.gz 2024-03-29T11:46:12.465 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T11:46:12.465 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34510.log 2024-03-29T11:46:12.465 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34470.log 2024-03-29T11:46:12.465 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.34510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34630.log 2024-03-29T11:46:12.465 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.34470.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:46:12.465 INFO:teuthology.orchestra.run.smithi089.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34510.log.gz 2024-03-29T11:46:12.465 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T11:46:12.465 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.34630.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34470.log.gz 2024-03-29T11:46:12.465 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34710.log 2024-03-29T11:46:12.465 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T11:46:12.465 INFO:teuthology.orchestra.run.smithi089.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34630.log.gz 2024-03-29T11:46:12.465 INFO:teuthology.orchestra.run.smithi089.stderr: 88.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:46:12.465 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.34710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34670.log 2024-03-29T11:46:12.465 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T11:46:12.465 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34710.log.gz 2024-03-29T11:46:12.466 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.34670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34590.log 2024-03-29T11:46:12.466 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-osd.5.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34670.log.gzgzip 2024-03-29T11:46:12.466 INFO:teuthology.orchestra.run.smithi089.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.34430.log 2024-03-29T11:46:12.466 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T11:46:12.466 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.34590.log: /var/log/ceph/ceph-client.admin.34430.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:46:12.466 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-mgr.x.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34590.log.gz 2024-03-29T11:46:12.466 INFO:teuthology.orchestra.run.smithi089.stderr: 87.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34550.log 2024-03-29T11:46:12.466 INFO:teuthology.orchestra.run.smithi089.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34430.log.gz 2024-03-29T11:46:12.466 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T11:46:12.466 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/ceph-client.admin.34550.log: /var/log/ceph/ceph-mon.b.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34550.log.gz 2024-03-29T11:46:12.466 INFO:teuthology.orchestra.run.smithi089.stderr: 87.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:46:12.466 INFO:teuthology.orchestra.run.smithi089.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T11:46:12.500 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-mon.c.log: 92.2% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T11:46:24.098 INFO:teuthology.orchestra.run.smithi072.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T11:46:24.190 INFO:teuthology.orchestra.run.smithi072.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T11:46:24.402 INFO:teuthology.orchestra.run.smithi089.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T11:46:24.451 INFO:teuthology.orchestra.run.smithi089.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T11:46:28.204 INFO:teuthology.orchestra.run.smithi072.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T11:46:28.264 INFO:teuthology.orchestra.run.smithi072.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T11:46:28.365 INFO:teuthology.orchestra.run.smithi089.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T11:46:28.482 INFO:teuthology.orchestra.run.smithi089.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T11:47:03.027 INFO:teuthology.orchestra.run.smithi072.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T11:47:06.333 INFO:teuthology.orchestra.run.smithi072.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T11:47:06.335 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-03-29T11:47:06.335 INFO:teuthology.orchestra.run.smithi072.stderr:real 0m53.939s 2024-03-29T11:47:06.335 INFO:teuthology.orchestra.run.smithi072.stderr:user 2m33.929s 2024-03-29T11:47:06.335 INFO:teuthology.orchestra.run.smithi072.stderr:sys 0m5.357s 2024-03-29T11:47:18.070 INFO:teuthology.orchestra.run.smithi089.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T11:47:18.072 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-03-29T11:47:18.072 INFO:teuthology.orchestra.run.smithi089.stderr:real 1m5.636s 2024-03-29T11:47:18.072 INFO:teuthology.orchestra.run.smithi089.stderr:user 1m57.004s 2024-03-29T11:47:18.072 INFO:teuthology.orchestra.run.smithi089.stderr:sys 0m4.330s 2024-03-29T11:47:18.072 INFO:tasks.ceph:Archiving logs... 2024-03-29T11:47:18.073 DEBUG:teuthology.misc:Transferring archived files from smithi072:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622722/remote/smithi072/log 2024-03-29T11:47:18.074 DEBUG:teuthology.orchestra.run.smithi072:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:47:42.545 DEBUG:teuthology.misc:Transferring archived files from smithi089:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622722/remote/smithi089/log 2024-03-29T11:47:42.545 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:48:01.347 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:48:01.571 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=a2101a5ba0744f31bb029b5767469427 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:48:01.575 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T11:48:01.584 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T11:48:01.612 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T11:48:01.612 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:48:01.627 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:48:01.706 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T11:48:01.716 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T11:48:01.717 DEBUG:teuthology.orchestra.run.smithi072:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:48:01.720 DEBUG:teuthology.orchestra.run.smithi089:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:48:01.733 INFO:teuthology.orchestra.run.smithi072.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:48:01.736 INFO:teuthology.orchestra.run.smithi089.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:48:01.738 INFO:teuthology.orchestra.run.smithi072.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:48:01.738 INFO:teuthology.orchestra.run.smithi072.stdout:=============================================================================== 2024-03-29T11:48:01.738 INFO:teuthology.orchestra.run.smithi072.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 1 +397us[ +397us] +/- 73ms 2024-03-29T11:48:01.738 INFO:teuthology.orchestra.run.smithi072.stdout:^* hv02.front.sepia.ceph.com 3 6 377 63 -122us[ -118us] +/- 68ms 2024-03-29T11:48:01.738 INFO:teuthology.orchestra.run.smithi072.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 62 -237us[ -237us] +/- 67ms 2024-03-29T11:48:01.738 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:48:01.741 INFO:teuthology.orchestra.run.smithi089.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:48:01.741 INFO:teuthology.orchestra.run.smithi089.stdout:=============================================================================== 2024-03-29T11:48:01.741 INFO:teuthology.orchestra.run.smithi089.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 61 +387us[ +387us] +/- 72ms 2024-03-29T11:48:01.741 INFO:teuthology.orchestra.run.smithi089.stdout:^* hv02.front.sepia.ceph.com 3 6 377 62 -120us[ -124us] +/- 68ms 2024-03-29T11:48:01.741 INFO:teuthology.orchestra.run.smithi089.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 62 -241us[ -246us] +/- 67ms 2024-03-29T11:48:01.741 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:48:01.742 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T11:48:01.751 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T11:48:01.752 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T11:48:01.792 DEBUG:teuthology.orchestra.run.smithi072:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:48:01.863 DEBUG:teuthology.orchestra.run.smithi089:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:48:01.933 DEBUG:teuthology.orchestra.run.smithi072:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:48:01.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:48:01.965 DEBUG:teuthology.orchestra.run.smithi089:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:48:01.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:48:01.997 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T11:48:02.006 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T11:48:02.033 INFO:teuthology.task.internal:Duration was 1309.357041 seconds 2024-03-29T11:48:02.034 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T11:48:02.082 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T11:48:02.082 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:48:02.085 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:48:02.128 INFO:teuthology.orchestra.run.smithi072.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:48:02.130 INFO:teuthology.orchestra.run.smithi089.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:48:02.340 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T11:48:02.340 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi072.front.sepia.ceph.com 2024-03-29T11:48:02.340 DEBUG:teuthology.orchestra.run.smithi072:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:48:02.361 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi089.front.sepia.ceph.com 2024-03-29T11:48:02.361 DEBUG:teuthology.orchestra.run.smithi089:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:48:02.382 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T11:48:02.382 DEBUG:teuthology.orchestra.run.smithi072:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:48:02.404 DEBUG:teuthology.orchestra.run.smithi089:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:48:02.454 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T11:48:02.455 DEBUG:teuthology.orchestra.run.smithi072:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:48:02.472 DEBUG:teuthology.orchestra.run.smithi089:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:48:02.592 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T11:48:02.603 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T11:48:02.603 DEBUG:teuthology.orchestra.run.smithi072:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:48:02.630 DEBUG:teuthology.orchestra.run.smithi089:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:48:02.657 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T11:48:02.666 DEBUG:teuthology.orchestra.run.smithi072:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:48:02.672 DEBUG:teuthology.orchestra.run.smithi089:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:48:02.695 INFO:teuthology.orchestra.run.smithi072.stdout:kernel.core_pattern = core 2024-03-29T11:48:02.722 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern = core 2024-03-29T11:48:02.744 DEBUG:teuthology.orchestra.run.smithi072:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:48:02.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:48:02.770 DEBUG:teuthology.orchestra.run.smithi089:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:48:02.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:48:02.799 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T11:48:02.809 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T11:48:02.810 DEBUG:teuthology.misc:Transferring archived files from smithi072:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622722/remote/smithi072 2024-03-29T11:48:02.810 DEBUG:teuthology.orchestra.run.smithi072:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:48:02.850 DEBUG:teuthology.misc:Transferring archived files from smithi089:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622722/remote/smithi089 2024-03-29T11:48:02.850 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:48:02.885 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T11:48:02.886 DEBUG:teuthology.orchestra.run.smithi072:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:48:02.888 DEBUG:teuthology.orchestra.run.smithi089:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:48:02.934 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T11:48:02.943 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T11:48:02.943 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T11:48:02.976 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T11:48:02.976 DEBUG:teuthology.orchestra.run.smithi072:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:48:02.978 DEBUG:teuthology.orchestra.run.smithi089:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:48:02.990 INFO:teuthology.orchestra.run.smithi072.stdout: 265507 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:48 /home/ubuntu/cephtest 2024-03-29T11:48:02.992 INFO:teuthology.orchestra.run.smithi089.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:48 /home/ubuntu/cephtest 2024-03-29T11:48:02.993 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T11:48:03.026 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T11:48:03.225 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T11:48:03.260 DEBUG:teuthology.nuke:shortname: smithi072 2024-03-29T11:48:03.260 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:48:03.303 DEBUG:teuthology.nuke:shortname: smithi089 2024-03-29T11:48:03.303 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:48:03.310 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi072.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622722', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:16:41.526377', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBnkRsNyU0uNAD+2aatvNpmr+GleVEkLFEE6m7MMj06nj5J8vftU4nbNo8dSce5aVYqvPCwgkVTxgpesC/Q1+bI='} 2024-03-29T11:48:03.336 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi089.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622722', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:16:41.527681', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPxSfkA8KNtwlNRpzeShqCxAH0NIYQ6PieBELRqnTCPw6tYsU9YdIh3hFkW56jYhLxj9fCJLD06xQQkl7q+9VVY='} 2024-03-29T11:48:03.380 INFO:teuthology.orchestra.console.smithi072:Power off 2024-03-29T11:48:03.380 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:48:03.416 INFO:teuthology.orchestra.console.smithi089:Power off 2024-03-29T11:48:03.416 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:48:03.429 DEBUG:teuthology.orchestra.console.smithi072:power off output: Chassis Power Control: Down/Off 2024-03-29T11:48:03.429 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:48:03.450 DEBUG:teuthology.orchestra.console.smithi089:power off output: Chassis Power Control: Down/Off 2024-03-29T11:48:03.450 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:48:03.467 DEBUG:teuthology.orchestra.console.smithi072:check power output: Chassis Power is on 2024-03-29T11:48:03.488 DEBUG:teuthology.orchestra.console.smithi089:check power output: Chassis Power is on 2024-03-29T11:48:07.468 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:48:07.489 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:48:07.579 DEBUG:teuthology.orchestra.console.smithi072:check power output: Chassis Power is on 2024-03-29T11:48:07.600 DEBUG:teuthology.orchestra.console.smithi089:check power output: Chassis Power is on 2024-03-29T11:48:11.584 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:48:11.601 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:48:11.695 DEBUG:teuthology.orchestra.console.smithi072:check power output: Chassis Power is on 2024-03-29T11:48:11.710 DEBUG:teuthology.orchestra.console.smithi089:check power output: Chassis Power is off 2024-03-29T11:48:11.810 INFO:teuthology.orchestra.console.smithi089:Power off completed 2024-03-29T11:48:15.699 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:48:15.810 DEBUG:teuthology.orchestra.console.smithi072:check power output: Chassis Power is off 2024-03-29T11:48:15.911 INFO:teuthology.orchestra.console.smithi072:Power off completed 2024-03-29T11:48:16.016 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} duration: 1309.3570413589478 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=a2101a5ba0744f31bb029b5767469427 status: fail success: false 2024-03-29T11:48:16.017 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:48:16.100 INFO:teuthology.run:FAIL